Skip to content

Is the precision with respect to the scalar value, or the formatted value? #17

@eemeli

Description

@eemeli

With mixed units, we're liable to end up representing e.g. 5ft 10in as something close to 5.83333, which when formatting needs precision to be defined, so that we don't end up with "5ft 9.99996in".

As with the scalar value that's discussed in #11, we need to determine whether the precision would be indexed by the smallest or largest unit:

  1. precision: 2 if by the largest unit
  2. precision: 0 if by the smallest unit

Even if we end up holding the value indexed by the largest unit, I'm starting to think that the precision ought to be measured as fractional digits of the smallest unit. Otherwise, we end up needing to come up with way too many solutions to corner-cases like what precision: 1 might mean in the above, or what precision: 2 means when formatting a value like "5ft 1in".

A key realisation here (at least for me) is that the total precision of a measurement should take into account the units: when formatting with foot-and-inch, there's a built-in expectation of the value including inches, which isn't present when formatting foot units. Therefore, the precision is most useful if it's considered to be going beyond the smallest declared unit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions