If code does the wrong thing and nobody uses it, is it really a bug?

A software bug is surprisingly hard to define. But it is something like "software that doesn't do what it's supposed to when used. Where 'supposed to' is some kind of documentation between a maintainer and a user." So the obvious case is when the code is documented to "produce X when given L", but when the user gives L they get Z, that is pretty clearly a bug. But that simple example is less helpful than we would like because the situation is normally more like a user gave it I (which they thought should work because it looks like a L under the right circumstances) and got Z. At least three situations can occur in the resulting bug report:

• The user can be convinced that L and I are fundamentally different. So no bug exists. The report should be closed with "won't fix".
• The maintainer can be convinced that due to the similarity of L and I the software should be changed to handle them consistently. So there is a bug. The report should result in a PR that fixes it with a test case covered by CI.
• The maintainer can be convinced that given the potential confusion between L and I the documentation needs to be fixed. There are many kinds of things that could fix this, perhaps the function in questions documentation should be updated, or maybe the entire libraries documentation (the readme should clarify that it works on L = Lennix but not I = Mac), or perhaps in error message should be added that triggers when used with I.

As if that wasn't complicated enough, now let's imagine we don't have the user do we still have a bug? One could object, that you can't have software with no users, because the maintainer also acts as a user. This objection feels reasonable. When I say, as I have in the past, "I found a bug in my unreleased code" I might mean that the code I wrote does not behave the way I expect when used the ways I have used it. In that case I am both the maintainer and the user of the code and it's clearly a bug. On the other hand I also find myself saying that when I have not yet had a chance to write any code using the behavior, or where the inputs on which the code would act unexpectedly are not ones I use. So I must mean something more like someone in the future who may or may not be me might use this code with these inputs and then would be surprised by the output. So it seems like there can be bugs that nobody has hit yet, as long as they are reasonably likely to do so in the future and would be very surprised by the outcome. This suggests that if integral(probability_of_use(J) * surprise_of_outcome(J) * dJ, for all J) is high enough then we have a bug even if there are no users.

Of course our potential users might be malicious. A malicious user might look for maximal points in the input to surprise_of_outcome and then intentionally providing those inputs, such that probability_of_use ~= 1 for suspicious enough outcomes. So security vulnerabilities are often in the situation of saying "I don't know why anyone would do this, and I don't think they have, but if you did the impacts are larger than you would expect". "Hacker provided input leads to code execution" is a bug because of how devastating consequences are, even if it would seem obvious that you shouldn't let the hacker control this.

"What is a bug?" Clearly depends on community. "UB if used wrong" is generally not considered a bug in C/C++ world, but is a bug/CVE in the Rust community. Developing reliable programs in Rust is significantly helped by this community norm, although enforcing that norm has occasionally devolved into profoundly unpleasant things. UB can cause such a breadth of unexpected outcomes, that I'm happy to work in a community where (even if intentionally daft inputs are required to triger it) UB is still considered a bug. Nonetheless, if a private method was incorrectly marked safe, but none of the public methods could be used to trigger UB it certainly doesn't qualify as a CVE. I'm not sure it qualifies as a bug.

It's not surprising that the definition of a bug depends on community, given that our definition of bug included some "user" who might be "surprised" by some behavior. A community is a group of people with shared expectations. Not very long before I joined the Rust community one of the clearly shared expectations was "on the latest nightly". People said "Here is an example of Rust code" but meant "that compiles on today's nightly". If you publish a library that claims to do FooBar everyone knew that meant "on the latest nightly", the maintainer was expected to release a new version when nightly change things, and the user was expected not to bother filing a bug report unless they were on the latest nights nightly. Through a series of harrowing fights there was a decision made to make some portion of language "stable" so that things wouldn't get out of date quite so quickly. When I joined (the 1.3 days) the existence of "stable" was settled but whether users had to care about stable was still up in the air. Many old-timers still implied "on the latest nightly" with everything they said and released, and were constantly annoyed by people who tried to run their stuff on stable. Over time as stable gained more necessary functionality, and people were tempted by the advantages of not having to keep up with the nightly treadmill so much, and people left the community the norms changed. Almost everyone who says Rust now means the Latest Stable Rust. Equivalently the community decided that "those crazy people who only wanted to run stable" were in fact part of the "supported community" so that their problems were bugs.

A community can only afford to support so many users at any given time. Hopefully it supports enough users that some of them become maintainers and the community becomes bigger and it can support more users. But if your community is going to be focused enough to actually support the users it has it needs to be very clear about the users it isn't (yet) able to support. For most of a decade (before 1.0) the Rust community was too small to afford to care about people who are on yesterday's nightly. As the community grew it could afford to care about people on nightly and people on stable. Some people wanted the Rust community to support highly regulated sectors, but the open source community was not robust enough to take on those users, luckily the people who wanted that support were able to fund additional people to focus on that effort. Some people wanted to use Rust on WASM, the community was not up for supporting those early adopters, so they added a target that claimed to be STD but would crash at runtime for the unsupported features. This let them experiment with this new technology without having to convince the entire community to support them. Turns out that was a very effective effort and now there are lots of people who use WASM most of whom wonder why it crashes that runtime instead of being a No-STD target. WASM is popular enough, and the Rust community is large enough, for a No-STD target and updating many dependencies to be a viable solution. (And people are trying to create such a new target.) Rust has grown so quickly that all the examples of growing the collection of supported users have grown the community enough to be huge successes. But that does not necessarily always hold. The pattern of fixing (potential) users problems faster than you are growing new supporters has killed other communities. Conversations about adding support for new community members are often fraught fights because it often sounds like providing less support to the existing users.

Another way the community focused its attention is by focusing only on the latest release of dependencies. The recommendation from the Cargo documentation was not to check in your lockfile (unless you're a binary) so that a significant portion of the community's attention would be focused on testing the absolute latest releases of dependencies. If you released a library, that had users, and you broke those users their next CI run would tell them so that they could tell you so that you could fix the problem. With how small the Rust community was, for a moderately popular project, the next time a reverse dependency ran CI might very well be a couple of weeks. If the community wasn't myopically focused on looking for new breaking changes they would never be found. As the community grew (and as CI got easier to set up and run) the number of reverse dependency CI runs also grew. For a moderately popular project if you have a breaking release you're likely to have several bug reports within a few hours. Dealing with the multiple reports often delayes getting they issue fixed. Also, the time the reverse dependencies waste waiting for you to fix your problem is also getting more expensive. The community large enough to no longer need to be quite as focused on the latest release. The Cargo Team has updated its recommendation to recommend checking in the lock file (if that's the right decision for your project), while having a CI job to test latest dependencies. Hopefully this will lead to fewer reverse dependencies wasting time while keeping enough attention on the latest version for at least one bug report to be posted promptly.

One of the interesting consequences of significant libraries being written in Stable Rust is that some of these libraries can be done. There are not new versions being released because there isn't new functionality that needs to be added. Because of the amazing work of the compiler team guaranteeing thatif your code works on Rust 1.K then it works on 1.(K+N) there do not need to be releases to keep up with the language. Building real code on The Latest Stable Rust was radical enough when I joined the community, but over time the possibility of building real software on Older Stable Rust is becoming more practical. The fact that it worked did not answer questions about if it is something the community should spend its time "supporting". The rust project itself only supports the latest compiler. If there are bug fixes only the current stable gets updated even if it was a security issue. But the compiler is an amazingly solid piece of work and security issues are very rare. So in practice building real software on an Older Stable Rust keeps getting better. Many conversations happened over the years without changing this uneasy equilibrium. Some members of the community, including very committed leaders of large projects, decided that if people were doing it they should support it. There projects documented that if there library didn't work on an older compiler you could file a bug and get it fixed. Much of the rest of the community continued to follow the compiler's lead and considers using the "wrong" compiler as the source of any problem and therefore do not consider it a bug. Cargo added a field in the Cargo.toml for specifying that maintainers do not want to be bothered with older compilers (by aggressively updating the flag) or that you're willing to support older versions (by leaving the flag set to an older value). This was a step forward, but did not change the equilibrium.

There has been a recent RFC to utilize Cargos resolver to help solve situations where the direct dependency supports your version of Rust but a transitive dependency no longer does. Specifically by making dependency resolution decisions based on the metadata set by the existing flag, and if needed leaving you on an older version of your transitive dependency. I could spend a few paragraphs going over the technical decisions but I fear I would never get back around to the point I was trying to make when I started writing. To summarize it was a long fraught conversation as often as not because of the underlying "who is in and who is out" and "If we provide support for more use cases does that mean less support for existing users" than the actual technical questions.

Where was I… So there's a flag in Cargo.toml to specify how old a Rust can be used with this project. If it's set correctly, then it's really useful information for communicating from the maintainer to the user under what circumstances the library is expected to work. But what happens if it's not set correctly? Well there are two possibilities. If the flags value is set too low, then there is a version of Rust that is documented to work but if someone tried to use it things would not work. That's a pretty open and shut bug. (Even if its consequences likely just a compilation error.) What about being set too high? Under this circumstance the library does not work with a users version of Rust, but would work with that Rust with little (or no) change to the other content of library. The library does not work on inputs it specifically documented it is incompatible with. That seems not to be a bug. The user might be annoyed, but they are requesting a feature not a bug fix.

Here's the other weird thing about this conversation, people who are using an old Rust and cannot easily update if they were reminded to are rare. They're not unheard of, and the use cases are getting more common. But significantly more human effort has been spent arguing about how to support these use cases than has been spent by people in these use cases. Our community has managed to get neither the benefit of making these use cases easy, nor the benefit of the community deciding to focus on other things. If everyone set the flag too low while making no effort to check that it was correct, almost all of them would have introduced unexpected behavior. There code would fail to compile even though it said it was compatible with an old Rust. But given how rare it is for someone to actually use a significantly old Rust and how small the consequences (usually a compile error) for most projects it would hardly qualify as a bug. Unless there were users of that project who did use an old Rust, who can file (an entirely legitimate) Bug report to inform the maintainer that they are one of the lucky ones who needs to actually take testing this functionality seriously. Perhaps this value of "I do not think about old Rust but might be willing to if asked" is best marked by a special value in the cargo.toml like the field being missing.

The same thought experiment could be run the other way. If the entire community set the cargo field to the latest stable, but were willing to lower it when there was a feature request to do so from a user who needed if for them selves. The community would rapidly learn which packages (and there would be some) need to support how far back. Other packages could move on without worrying. If someone needed them to lower the flag they could, until then why worry. Maybe we should have a special value for "I stick to the latest stable because I haven't thought about it too hard but might be willing to support more if asked" marked by a special value like toolchain. There seems little downside to automation that sets the field conservatively to a value that is always at least as high as it needs to be. This automation would induce some feature requests but never introduce bugs. But perhaps I am having too much fun with semantics. If the automation prevents things from working that would have worked if the automation had instead done nothing, then it's making people's lives harder even if this literalist could claim it's not a bug.

So after this navelgazing literalist has had his fun, where are we at? First off, we should relax a little. The hard questions here are about what the community is willing to support and Teams/decisions/RFC's/Tooling has only limited impact on the community. Given how fast the community is growing, it is likely to be willing to support more over time. Next, we should improve the tooling primarily Cargo's resolver to make it easier for users to just get their job done, but also Clippy, Rustc, cargo-semver-checks, and cargo-msrv To make it easier for maintainers to keep the MSRV metadata correct. Last, we should try and focus on users who demonstrably exist and not spend significant effort attempting to support users who potentially could exist. Given the odd dynamics of the situation, we have the potential to "Spend less effort" or "Support more users" pick two.