First of all, it's in 3/4. Because it's a waltz, you see. And waltzes are in 3/4.
Within this dance lie a handful of little touches that elevate "Waltz #2 (XO)" to its greatest potential. The verses are long, but the chorus is only two lines, which makes it feel less like a chorus than a refrain. The refrain is: "I'm never gonna know you now / But I'm gonna love you anyhow," which is one of those lines that I'm realizing I love because it's weighty without being particularly complex.
The piano line seems to chase the vocals around the song, like a child chasing a cat, and halfway through the second verse the piano steps aside and we get an organ interlude while Elliott Smith's singing "You're no good / you're no good / you're no good."
But what I love most about this song is its sense of forward motion. Calling it a waltz and backing it with that drum part definitely go a long way toward giving off the impression that it's moving its feet. The verses exist in units of four measures, and the last two downbeats of the fourth measure really get stomped on. For 10 beats we're just strolling along, and on beats 11 and 12 there's a gap in the sidewalk or something and we pick up and hop over the cracks. The end result is that a song about despair and unhappiness--which, it's Elliott Smith, so I don't know what you were expecting--has a very palpable sense of joy.