11
$\begingroup$

I am a developer in the machine learning team, and mostly involved with the neural net repository. One of the discussions with my group raised these question in my mind, and these might be stupid questions, but here they are:

Question 1: Would our users rather want more models in their respective field of applications, that are correct, functioning and tested, or would they want models that are correct, functioning tested and properly structured? In other words, a lot of the times, models when imported are not structured properly, and many a times we spend a great deal of time restructuring them (in lack of better words, prettifying them).

If I were a user, I would rather have more models that are correct and ugly, rather than less models that are correct and pretty, because a great deal of developer time is spent in "prettifying them". However, the question to the users here, is there any potential benefit of having a pretty model as opposed to an ugly one, that could justify endless developer hours being spent on making a model pretty? I maybe missing some details or use of pretty and structured models here, and that's why I reached out to our community to see if there are any advantages to justify the tradition of making the published models pretty. This would potentially help us free some time to actually convert models for various application areas.

Here's a more concrete example of pretty v/s ugly: https://drive.google.com/drive/folders/1bnrN4xELVGNObxwSMrnvijBcm_bU2V0P?usp=sharing

Please import the models to learn about the pretty and ugly versions (there are other differences). The .onnx has only the backbone and the FPN (you can verify that the output are the 4 probabilities and box locs). A first-level restructuring where relevant sections are marked backbone, FPN and the rest of the sections would not take much time, but to prettify it to the full, would require a lot of restructuring. For practical purposes, one would simply want to extract the relevant sections like backbone and/or BiFPN for further uses.

Question 2: That also brings me to the next topic, how important are each of these sections in the main page you see for the models in the WNNR? https://resources.wolframcloud.com/NeuralNetRepository/resources/EfficientNet-Trained-on-ImageNet-with-AdvProp-and-AutoAugment/ e.g. If someone could rank the usefulness of each section of the above shingle page, starting from "Resource Retrieval" to "Export to MXNet" that would be great as well.

Question 3: Would you prefer seeing more sections in such shingle pages? Sections explaining the architecture, like they do in the original research papers? If so, any ideas for how it should look/be formatted? How much details should each shingle page have?

P.S. All the groups tagged here are relevant application areas of the WNNR.

$\endgroup$
5
  • 1
    $\begingroup$ For "prettifying" do you mean structure the model construction to meaningful blocks as you do in the ConstructionNotebook? Can you give an example of "ugly" vs "pretty"? $\endgroup$ Sep 6 at 18:41
  • $\begingroup$ Crossposted here. $\endgroup$ Sep 6 at 18:44
  • $\begingroup$ Yes, for example if you open an ONNX graph in a viewer, it appears to be kind of messy, however you can see them have a structure in WL. While re-structuring some blocks do not take much time, however, for more complicated graphics like you would see in object detection, this practically means reconstructing the entire network by hand. For any practical purpose though, you can do surgeries the relevant parts, even without the "prettied" subparts. $\endgroup$ Sep 6 at 18:45
  • $\begingroup$ Here's a more concrete example of pretty v/s ugly: drive.google.com/drive/folders/… Please import the models to learn about the pretty and ugly versions (there are other differences). The .onnx has only the backbone and the FPN (you can verify that the output are the 4 probabilities and box locs). A first-level restructuring where relevant sections are marked backbone, FPN and the rest of the sections would not take much time, but to prettify it to the full, would require a lot of restructuring. $\endgroup$ Sep 6 at 19:05
  • $\begingroup$ I prefer a greater variety of "turn-key" deep nets, for a wider range of tasks. I think most users play around with lots of networks before they move into developing their own. I'd also like a cleaner interface for training with our own data sets. Another annoyance: OpenPose (to take one example) works with v. 12 but not 11.0. Can you make these networks more back compatible? $\endgroup$ Sep 6 at 19:34
8
$\begingroup$

I believe I'm a heavy user of the neural network framework and the Wolfram Neural Network Repository(WNNR) and these are how I see the WNNR from a user's perspective. Hope it will be helpful.

First of all, I think the developers are doing a great job putting together the neural network framework and the WNNR. The framework is well designed and easy to use. The documentation is concise and informative. I found many times I refer to the documentation and WNNR to understand a deep learning concept or a network model.

In my opinion, WNNR is a great extension of the neural network framework. The framework itself is concise. But perhaps because of the computational limitation of the documentation within Mathematica, examples in the documentation are often too simple and considered as toy problems. When applying the framework to real-world problems, there is often a big gap from these toy examples. And I think that's where WNNR comes in place. WNNR basically closes that gap to some extent, as most of the WNNR models are from publications that try to solve real-world problems. I personally use the ConstructionNotebook option very often to get more understanding of how a deep learning concept is implemented in real systems.

Let me give a concrete example using the AttentionLayer. In the documentation, the only example of AttentionLayer is about sorting 6 single-digit numbers. However, more practical usage of multi-head attention (for example in transformers) is not documented. With the help of the WNNR, I can just look up how it is constructed and even reuse part of the source code in the construction notebook for my own problem. Without a structured notebook, I will have to resort to the original paper and other python implementations for the details. With the structure and "pretty" of the models, WNNR takes much of the burden away from the users.

Moreover, I think reusability is more important than scope coverage. I found that it almost never happens that a WNNR model solves exactly the problem I have on hand. There are always some modifications I need to make to models. I would personally prefer a structured model that is easier to understand and modify.

In conclusion, I can imagine that using an automatic system to load the unstructured model to WNNR may require less manual effort, but I think a large portion of the values of the WNNR lies within the structures of a model because it bridges the gaps between the well-designed framework that are clean and concise and practical problems that sometimes are messy. The structure or "pretty" of a model not only extends the documentation of best practices approach with real-world problems but also provides reusable structures in terms of code and distilled knowledge.

$\endgroup$
8
  • 2
    $\begingroup$ The ability to re-train networks is the most critical thing. Construction notebooks training details are often empty. Nobody wants to just run models, they want to fine tune them! And usually that is totally non-obvious here. $\endgroup$
    – M.R.
    Sep 7 at 22:13
  • 2
    $\begingroup$ I agree, the ability to re-train the networks would be the ultimate piece to close the loop. I suspect most of the converted networks are not tested with training abilities and their weights are simply from the published trained networks from other systems. I think the best is that for every model WNNR can reproduce the model construction, model training, and model evaluation all within the Wolfram system rather than just loading the weights trained in other places. However, I can see that it would involve a lot of work to get the training work for most of the models. $\endgroup$ Sep 8 at 1:18
  • 1
    $\begingroup$ I agree that most of the models are probably too large to train from scratch and there is probably no need for that for most users. I think ultimately, it is about reusability, i.e. whether it's easy to adapt the model to a specific problem/dataset. A fine-tune section would definitely be very helpful. I can see that there is a lot of work involved in importing a network to WNNR and I think the developer is currently doing a good job. $\endgroup$ Sep 9 at 3:29
  • 1
    $\begingroup$ My motivation of writing this answer is that I was concerned that those "prettify" efforts will go away and a future converted model would be a spaghetti graph presented using the low-level MXNet operators (like what you get by importing the onnx model), which would be much less useful for the users. I'm also wondering whether wolfram is considering the possibilities of open WNNR for user contributions, just like wolfram function repository, which may greatly accelerate the growth of the WNNR. $\endgroup$ Sep 9 at 3:29
  • 1
    $\begingroup$ For your concern about not prettifying - we will absolutely not just import the ONNX models and leave it as the low-level graph that you would see. The fact that we go over the architecture, and restructure them according to the original paper and/or sections that can be easily performed surgery on. I agree having a ONNX Import dump would not be aesthetically pleasing and/or user friendly. So your concern is well received. However, prettifying to the lowest level possible takes a lot of effort, so the question was if a middle ground is still acceptable. Maybe in the process I can share details $\endgroup$ Sep 10 at 2:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.