The state of systems research has improved significantly in the last five years. If I see SOSP/OSDI papers in 2011 or 2013, it was rare to have open-source code. If you look at papers published in OSDI/SOSP in the last few years, it is rare to *not* find open-source code.
-
Show this thread
-
Replying to @vj_chidambaram
this feels exaggerated and it would be nice to see some data. microsoft release more open source for sure.
1 reply 0 retweets 1 like -
Replying to @xexd
I am not saying Pre 2011 people didn’t release source code at all, just as even now fb and other companies don’t release code for sosp/osdi papers. But I definitely do see a trend towards more open sourcing now, especially from academia. I’ll try and get some numbers when I can!
1 reply 0 retweets 0 likes -
Replying to @vj_chidambaram @xexd
I thought this was a trend everybody was seeing though. I’m significantly less confident now if you are not seeing this as well. I’ll get the numbers and see if I was mistaken.
2 replies 0 retweets 0 likes -
Replying to @vj_chidambaram
“improved significantly” and “rare” aren’t well-defined so we can both be right. i do know from doing both that open source is not the same as research, and don't think source code release has much to do with “the state”
1 reply 0 retweets 1 like -
Replying to @xexd
The connection is empirical comparison to prior work. If the source code isn't available, and the system is complex, all that can be done is a principle/technique-level comparison. I think being able to run and compare against prior systems makes for more rigorous science?
1 reply 0 retweets 0 likes -
Replying to @vj_chidambaram @xexd
For example, I review some papers where authors claim they are better than some prior system. If prior system is open-source, I simply ask for them to evaluate against it. For older complex systems, it becomes much less clear if they would be better or not.
2 replies 0 retweets 0 likes
IMO it really depends. Could just be engineering/implementation differences that are irrelevant to the core design/ideas. An open-source implementation may help understand the paper but evaluation directly using these implementations can also be misguided.
-
-
Replying to @bipartite @xexd
Yup, definitely agreed. But all things considered, I’d rather have an evaluation with the prior system being run rather than one without.
0 replies 0 retweets 0 likesThanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.