I think there are two reasons why people start collecting records.
The first reason is that it's an expression of admiration. The simplest case is, of course, that of the teenage girl who buys everything by, say, Take That or Justin Bieber (or whoever it may be at the time), just to feel close to the star she fell in love with. I don't think that the "serious", grown-up record-collector who concentrates on the music of a single artist is much different from that poor teenage girl who got lured into the nets of the music industry. Ultimately, any collector had once fallen in love with his preferred star, albeit in a more abstract sense, by, say, having been attracted to the style (musically and visually), voice, song-writing or whatever. Admittedly, there are differences to the teenage girl - never in my life have I wanted to kiss David Bowie! But then again, paying out large of sums of money for some re-issue box set that contains only music I have known and owned for decades is not too different from what the Justin Bieber fan does. By and large, it's the same expression of admiration, even if the reasons might be more complex.
The second reason for starting a record collection (regardless of whether it's a collection of a particular artist, style or a "general" collection of everything you like) is the natural human desire to have something that your neighbour hasn't got. Most people satisfy this need by reverting to more obvious things: the bigger car, the bigger house, the more expensive watch, the cooler smartphone, or, if you're in the advanced category, the bigger yacht. Record collectors are not very different, in some sense - for good reason the punk band Poison Idea called their 1984 EP Record Collectors Are Pretentious Assholes!
The problem is that there is hardly anyone you can impress with your collection when you're beyond your teenage years - I don't know why, but the car/house/watch/smartphone/yacht solution seems to work better... But maybe that's the reason why I like collecting records: you're not part of that embarrassing treadmill of heaping up socially accepted status symbols. The other day I told some colleagues that I had to visit the customs office to pick up a parcel from South Africa. If I had told them that it contained an iPod I had bought there because it was cheaper in South Africa, they probably would have understood. But when I told them that it was a record, I just got incredulous looks: "You bought a record from South Africa?" I like collecting records. I guess I'm a pretentious asshole...

A Personal Account of Collecting Bowie Records
(or: "How to Spend Your Money on Things that No Rational Person Needs")

When you start a collection of things of any kind you rarely have a plan that you follow: you just buy the things that look "interesting" to you. And, to be honest, I've never got beyond that stage.
I started as a fan, that is, as someone who wanted to have as much as possible by his artist of choice. And when I started collecting Bowie records at the tender age of 13 (in 1980), I didn't know anything about originals and re-issues, label designs, catalogue and matrix numbers, etc. - I just bought any record that had the name "David Bowie" on it. I think that's the way most record collections of a particular artist start. At that time, a different record was just something that had a different cover. At this early stage, this view was supported by the meagre discographies in such books as the German issues of Claire and Douglas. Those were fan books, and not particularly good ones...
The time I'm talking about are the early 1980s. In those days, there was nothing like eBay or so. "Serious collectors" (who were older than me) went to record fairs or purchased their records via magazines like the German Oldie-Markt or its UK and US counterparts Record Collector and Goldmine. Neither of these possibilities was in the reach of a poor teenager like me in the early 80s.
Instead, my primary source of records were the regular record shops and the "famous" German flee market of Hanover - in fact, it does have a good reputation in Germany! Though certainly not one comparable to Kensington Market... Lucky UK fans of Bowie!
There were a number of remarkable events that changed my view of collecting Bowie records. The first one was when I leafed through some Bowie book in a record shop in, say, 1981, which had a list of "David Bowie's unofficial records". I still wonder what book that was. At the time, I didn't have the money to buy it, of course. I had thought that I had an (almost) "complete" collection of Bowie records (re-issues, that is), and now I had to find out that there were bootlegs! Some time later I actually came across my first Bowie bootleg in real life (at the Hanover flee market). I think it was some edition of The All American Bowie. And I didn't have the money to buy it! So I decided not to let such an opportunity go ever again, saved all my pocked money, and during my next visit to Hanover I bought Golden Years of Bow and Live in Bremen '78. I felt like being part of an elite!
Another important event was a visit to Amsterdam in 1982 (at the age of 15). In those days, there were numerous little record shops in Amsterdam, and all of them had different Bowie singles in different covers. I had to realise that there was no such thing as, say, the Boys Keep Swinging single, but that there were countless different issues of that single... I didn't have the money to buy them, but I bought Carr & Murray's An Illustrated Record instead. Not only was that the first serious Bowie biography I read, but it also showed me actual pictures of Bowie records which I would like to have had, but had never thought to be within my reach (like, say, the dress cover version of The Man Who Sold the World). It wasn't before 1989 that I bought my first genuine Bowie record from the 1960s! It was a battered coverless German copy of Space Oddity - which has, of course, been exchanged for a better copy since then.
Throughout the 1980s I kept collecting Bowie records that way. I bought all new stuff, and often (overpriced - from today's point of view) second-hand records. However, a lot of money was spent on the music of other artists - there's more to life and music than David Bowie. In the early 1990s I temporarily lost interest in the contemporary music scene (due to domination of Grunge/Hip Hop/Techno and the decline of the vinyl record). At that time I started my collection of 1960s records and re-discovered the joys of collecting Bowie stuff. At the same time I began to visit record fairs and buy records by mail-order. Especially the German Oldie-Markt (an auction-per-mail magazine, only roughly comparable to the British Record Collector and the American Goldmine) was a useful source to expand my collection. In addition, I had more money than in my teens, and so I could buy things like my copy of the Philips album or the US mono issue of the debut album. So to speak, the 1990s expanded my view on collecting records from a local one to a nation-wide one. Things changed again with the rise of the internet from the early 2000s onwards. International record-dealers and, especially, eBay have made collecting records a world-wide affair. Suddenly, it was possible to find and buy records in Japan, and pay for them within seconds - in the 1990s you had to send cash in a letter! Very often these letters got lost...
I still lose interest in Bowie from time to time. There are times when I think there's nothing more interesting than getting a copy of some Bowie record with an unusual track listing, cover, label design or even catalogue number. And then there are times when I think there's nothing more boring than that. Which extreme it is depends on various factors, such as how enticing the contempary music scene is, what I'd like to spend my money on, or what other things keep me busy (for a while, I've planned a website on the history of rock music in general, but I've shelved that... for the time being). But somehow, I can't resist falling back on Bowie from time to time... But, with a whole planet full of Bowie records within my (virtual) reach, what do I collect?

On Completeness

Bowie hasn't been - and isn't - the only artist whose records I collect. For example, I collect the records of Paul Roland, and it is my aim to have every vinyl record by him. I haven't achieved the feat, but at least my collection of Paul Roland's records is almost complete. I can't say that about my Bowie collection...
The first fact that you must face when you start collecting records by a "major" artist like Bowie is that your collection will never be complete. You could decide to concentrate on all records from a certain country, but when you've got all regularly issued records, you will find out that there are also promos. And when you've got all promos, you can start buying all the bootlegs. And, of course, you will like to have at least the more interesting records from other countries - that is, those with tracks that are not available in your country of choice. And you will also probably want to have the more interesting cover variations. And so on...
Even if you managed to get a copy of all regularly released Bowie records in the world (including promos), you could still extend your collection by "unique" items, such as acetates. And, by the way, what is a "record"? Do you want to buy vinyl records or CDs, too? And what about DVDs, cassette tapes, VHS tapes, eight-track cartridges, and so on? And then there was this unique reel-to-reel tape with unreleased stuff from some recording session that I've recently seen on eBay...
So a "complete" Bowie collection must either be illusionary or limited in scope to such a the degree that it would be boring. Of course, you could collect Icelandic Bowie records, but your collection wouldn't be very interesting... I'm not entirely sure, but I think you would be among the thousands of people whose only Bowie records are the Let's Dance and Tonight albums! So don't aim at completeness - the only thing you can do is set priorities.

On Vinyl and CDs

Even a brief glance at my collection will show you that I prefer vinyl records to CDs. Bowie's recording career started when vinyl records were the only serious way of distributing records, and now we're in the age of CDs and digital downloads. When I started collecting Bowie records, vinyl was still the leading medium. CDs assumed that role in about 1990, but I have accepted them only reluctantly.
I don't want to revive the old battle between vinyl-lovers and CD fans. Maybe the sound of a vinyl record is "warmer" (it sometimes seems like that to me), but all in all, the sound of CDs is superior (unless they are re-mastered for maximal loudness - but that's a different problem). Moreover, they are much more convenient in everyday life, which is why I have recorded all my vinyl records to CD for everyday listening. So why don't I like CDs, nonetheless? That's where the collector comes into play again.
First of all, there is the issue of durability. A vinyl record (just like its predecessor, the shellac record) is made of a single material. Stored under ideal conditions, an analogue record can be playable for centuries. Shellac records from the beginning of the 20th century are still playable without any problems! The paper labels might get lost, but the labels could never damage the playing surface. CDs are different: they are made of multiple layers of different materials, which might - and certainly will - react with each other in the course of time. I wasn't surprised that the horror stories from the early days of the CD (spread by vinyl aficionados) turned out to be wrong: only very few CDs from the early 1980s have become unplayable by now, because inappropriate varnish had been used for the labelling. All my CDs still play perfectly. But by and large, it's only a matter of time. Even the producers of CDs assume a lifetime of about 50 to 100 years. You could now argue that this shouldn't bother me, because by the time my CDs will start to dissolve I will certainly have dissolved for decades. But I just hate the idea of paying out large sums of money for things that will probably turn useless within less than two or three generations. I don't care who will own my Bowie records when I'm dead and gone, but I hope he or she will appreciate them, and not just throw them into the nearest garbage can because they have become useless!
Second, each vinyl record is an "artefact" in its own right. It can be in mint condition, it can be in very good condition, and so on. The sound quality might have deteriotated, but the music might still be enjoyable. No, I don't want to support the legend that vinyl-lovers think that clicks and crackle are an integral part of a satisfactory listening experience! That's rubbish: every decent record collector dreams of having all his records in mint condition, and the first thing I do when I record a vinyl item to CD is filter out clicks and crackle. I don't need them for a good listening experience!
However, the mere possibility that there might be clicks and crackle makes each vinyl record a unique artefact. A record may be in practically perfect condition, it might be in mediocre or even bad condition. In a sense, the condition of a record reflects its unique history. CDs are simpler: either they play or they don't. Once a CD has a serious damage (such as a major scratch), you can only throw it away, because it's no longer playable at all. What's that compared to a vinyl record? My copy of the All the Madmen single has even got a crack through most of the playing surface, but plays great with only a minor click where that crack is! To sum up, it's the uniqueness of vinyl records that attracts me to them. To me, collecting vinyl compared to collecting CDs is like collecting paintings as opposed to collecting art prints. I guess you don't want to hear my opinion on "collecting" MP3-files...
But all that is just a subjective point of view... It's not my job or aim to convince or convert anybody.

So What Do I Collect?
(or: "Could you get to the point, please?")

As you might have expected, I primarily collect vinyl records. At one point of time I decided to buy a set of the original RCA CD issues of Bowie's "classic" albums, but I usually buy CDs only if they contain something that is not available on vinyl. Okay, if I have got a CD in my collection, and I come across a vinyl issue, I'll keep the CD nonetheless. That's the reason why my CD collection of Bowie stuff is very incomplete, and even quite common compilations are missing. I'm simply not interested in them, if they contain only familiar stuff.
Apart from that, I simply collect everything that looks interesting to me. Interesting is, of course, everything that contains music I haven't got, but also records with an unusual tracklist, records that are particularly old, rare or "exotic" (from my Eurocentric point of view). Bootlegs, picture discs, coloured vinyl issues, promos, unusual covers and even label variations that seem interesting to me. Ultimately, there is no real system. I'm not very interested in remixes, especially those dance-oriented mixes that have turned up on numerous (often semi-legal) releases in the past ten years. To me the mixing of a record is part of the creative process itself, so re-mixing a record is a bit like, say, overpainting parts of a work by your favourite artist. Admittedly, it sometimes works - there are examples of good remixes, which shed light on an interesting aspect of the original work - but in most cases, the results simply sound awful to me.
So there's nothing complete here. I think I've got some original copy of every German Bowie record that has been released until the early 1990s. But even my German collection isn't really complete: some of the EMI re-issues of the "classic" albums are still missing. At the time I preferred to use my limited budget to buy the American Ryko issues on clear vinyl.
So I just look here and look there, and buy whatever seems attractive to me. That's all...