On Data Science, Part I: Entering the Field and Avoiding Niggerdom

I always enjoyed helping students, undergraduate and graduate, get jobs. Most professors have never been gainfully employed and send students (out of disinterest if not incompetence) to equally clueless ‘academic advisors’ and ‘placement directors’. Now that I’ve left the professorate, one of my proudest accomplishments is having helped several people— on Twitter and in real life— secure nonacademic jobs, and by popular demand I’ve decided to share whatever wisdom into the process I can. I am writing this mostly with graduate students in the social sciences and related fields who want to pursue a career in ‘data science’ in mind.

With that said, I cannot improve on what Shakoist writes on ‘business data science’ in this post: Why Business Data Science Irritates. You should absolutely read this post if you’re considering a transition to industry data science. My perspective differs only by experience. While Shakoist is a gigabrain who works in the top of the tech world, I am a retard who juggles numerous jobs and turned down the only FAG (Facebook, Apple, Google) job offer he received. My only disagreement is you can find— usually by happenstance— the perfect ‘scientist’ job but it comes with drawbacks that I will explore later (mainly, you will be paid less).

A Typology of Data Jobs

You type ‘data scientist’ into LinkedIn or Indeed for the first time and are bewildered by the variety of companies, job descriptions, skills, compensations, and even job titles. Where do you start? I propose that you can, without dramatic loss, disaggregate the world of ‘data science’ positions into three types: Data Astrologer, Data Janitor, and Data Nigger.

Two points to start. First, there is no value judgement in these conceptual containers. Most academics would, in fact, make excellent Data Niggers. The reason you, dear reader, would probably detest such a position is your desire to escape academic serfdom and accomplish something greater. PhDs who would make the finest Data Niggers generally do not self-select out of academia, preferring to serve as an adjunct, visiting assistant professor, or other temporary academic role (or administration) in perpetuity rather than accept the perceived risk (or affront to their academic identity) of leaping into industry. I assume you desperately want to avoid falling into the position of Data Nigger, not because it is inherently undesirable but because it doesn’t suit the professional aspirations of whoever lands on this page.

Second, these are Weberian ideal-types that any single job can at best approximate. Most data-centric roles in industry involve some combination of astrological, janitorial, and data nigger work. Nevertheless, as you progress in your career your work is likelier to collapse into one category.

Data Astrologer: If you’re recently an academic, astrological work is most likely what maps onto your ‘data scientist’ mental image. It involves the careful development of Kabbalistic instruments called ‘statistical models’. What purpose these instruments serve varies tremendously. Social scientists will be pleased to hear sometimes this includes experimental and econometric work, although primarily of the “are customer likelier to open blue or orange dildo ads?” variety. Fancier rituals that discern what complicated astrological combinations augur are currently more in vogue and, at the highest level, the alchemists have learned to transmute digital symbols into infinite big-tittied anime girls. If you are, say, an economist or sociologist by training this highest order is withheld from you for a few years, since social scientists are currently untaught in these alchemical arts. However, schooled as you are in the first Aristotelian element— regression— you can with dedication learn the subsequent two: classification (logistic regression) and neural networks (polynomial regression).

Data Janitor: A janitor maintains, cleans, and repairs. Sometimes he writes a fifteen thousand page novel in the janitor’s closet because nobody bothers him; other times he builds a ladder extension with duct tape and stolen copper so he can smoke an apple bong on the school roof. Janitors are humbler, by nature of their work, than astrologers but more essential to organizational health. Anyone who comes from a scientific computing background and works with limited, canned datasets is initially culture shocked by the data in industry, not due its ‘size’ so much as its management. Data is housed everywhere: on multiple company servers, on five different cloud systems, on hundreds of scattered CSV files, on Pradeep’s OneDrive, on another company’s AWS account to which nobody on your team has access. Data Janitors make all the data run on time. If you’re an Astrologer, pray you have competent Janitors to assist you, in which case you’ll never complain that the Janitors make more money than such an enlightened soul as yourself. Not only is this instrumental for receiving appropriate data, but occasionally an upper-management bimbo will want your fancy tensorflow model deployed on Salesforce because they like that its homepage is the color blue. Data Janitors can help.

Data Nigger: Astrologers and Janitors both have autonomy. Shamanistic ritual gives the astrologer his autonomy. He performs incantations nobody else can fathom and, while he is often tasked with translating his arcane sorcery into plain language, even the executives are wary of dissent from fear of eliding syllables from these ancient spells. Janitors have autonomy because they are scarcely needed when all is in order. Called upon to fix mishaps, the janitor is free in the remainder of his time to pursue whatever experiments he might, so long as it doesn’t culminate in an FBI investigation. Data Niggers, in contrast, are characterized primarily by their servitude, their subordination. At the lowest level you have the cotton pickers, the Field Niggers, the data labelers who sits in dingy offices in Bangalore and Quezon City bobbling their heads in irregular directions while they spend 12 hours a day figuring out if a picture has a car in it or a text a racial slur. Now you, as a highly credentialed US worker, will never have these tasks; but in addition to field niggers there are also House Niggers, one of the more densely populated slave roles in corporate America.

Here is the catch, both astrologers and janitors can unwittingly become niggers. Each transmutes in its own special way. For the Data Astrologer, you know niggerdom is nearby when you hear speak of KPIs, which is business jargon for the lucky numbers inside fortune cookies. Executives are anxious to accumulate KPIs, for otherwise they would have to communicate to themselves and investors in natural language, logic, and branches of mathematics that use letters instead of numbers what the business does. Naturally this experience is frustrating for everyone because Netflix keeps releasing new original queer-dramas. It is inescapable that the Data Astrologer will at times work hastily to ensure KPI production. However, often this turns into pathological behavior and the organization exists solely to churn out evermore KPIs— we may as well call these Fortune Cookie Factories. In the Fortune Cookie Factory your time is so consumed in compiling KPIs into Excel workbooks and Powerpoint slides— never written documents, because nobody will read them— that there is scarcely time to utter the ancient incantations. Often in these situations the Astrologer is assigned a VP of Astrology, whose job is to hasten KPI production. Perhaps you need three weeks to sacrifice a goat and resolve the ‘blue dildo / orange dildo’ question as decisively as the Gods will allow. Your VP of Astrology will most likely decide you have three hours to instead run a forty-two variable regression because she read an article with that approach in 1993 and, regardless, that’s what she has always done. Then she waffles over the appropriate Excel formatting, whether forty-two is an unlucky number and should be forty-four, whether the Powerpoint slides should be blue and orange like the dildos, whether negative numbers actually make sense or are blasphemous—

Data Janitors have a more straightforward path to niggerdom. Under normal conditions, an organization runs smoothly enough to keep the data running on time without becoming onerous for the janitorial system. Like some schools, however, some businesses are in a state of prolonged and irreversible degradation and mere zombie-like survival demands constant, tedious maintenance just to avoid flooding or Salmonella poisoning. Just as with some schools, eventually the decay sets in motion a feedback loop where the worst self-select in and the talented run for the hills, and through incompetence the organization breaks more and faster than even a competent janitorial staff can keep up with. Eventually the Data Janitor notices all the new hires on his team are named Sancheerp and Androop; not the Stanford Sancheerp, but the mostly unintelligible, ‘how do I get out of this handshake’ Sancheerp. Very quickly the Data Janitor is called to every classroom in the building, tasked with 50 hours a week of mopping up bodily fluids that have no business existing in the public sphere outside wartime.

The Psychology of Niggerdom

Just as there are three ideal-typical data roles, there are three corresponding company types: the Startup, the Mediocre, and the Big Gay Zombie. Once again, these are ideal types and most companies do not neatly fit into these categories. Some companies try to emulate the startup environment in small cells segmented from the broader Big Gay Zombie (but, be warned: every ‘data science’ team in a Big Gay Company calls itself a ‘startup in a big company’ and most times it’s a complete fabrication).

Startup: I have worked at three startups and, in this limited experience, they share similar qualities. All offer a high degree of autonomy and are relatively laid back. One shouldn’t confuse this for an environment that rewards laziness. On the contrary, these are often the most intense environments: the problems are novel or challenging, your colleagues are the smartest and most effective, the company lives or dies on its ability to beat its competition. What makes this environment exceptional is you’re encouraged to utter your incantations with limited oversight. Unless hired into junior or support roles, where others guide you through the inner degrees of Freemasonry, the worst signal you can send in startups is helplessness. Everyone in these environments is exceedingly helpful, not in small part because everyone is an nth-level Astrologer, but none of them wants to be encumbered by Data Niggers who are paralyzed by the thought of independent problem solving and suffers from dreadful indecision about whether they ought to present 95 or 99 percent confidence intervals. It is not that they are unwilling to discuss the complexities of these topics— they love this— but they naturally intuit the Data Nigger psychology, the toady, the A student not because he understands the material but because he knows ‘what the teacher wants to see.’ Data Astrologers thrive in complex, novel, ambiguous settings, wholly unlike the Data Nigger who starves without a structured, micro-managed environment and develops obsequious traits as a consequence (or the reverse, by that point).

Mediocre: I have worked at two mediocre companies and contrary to the pejorative label these are also excellent ones to begin your career. Mediocre companies have reached a stage of maturity beyond the startup yet are not entirely sclerotic like the Big Gay Zombie. These are relatively stable, but unexceptional, companies where the primary function is to ‘keep the trains running on time’, rather than to experiment as you might in the startup world. All the Data Janitors I met inside mediocre companies have been quiet content. Ignoring the selection bias involved there, I believe one reason is that janitors thrive when they have relatively defined janitorial duties that do not occupy them to the point of exhaustion. They are integral to ‘keeping the trains running on time’ and are hence afforded a great deal of latitude to pursue side interests and projects. As a more astrologically inclined person myself, I find these environments a bit unexciting as the work tends toward janitorial functions over time, but like many other former academics I have come find janitorial work satisfying. Data Niggers dislike this environment for an entirely different reason, ones that require attention to the inverted hierarchy central to the Data Nigger’s fragile self-representation.

Zombie: Zombies are megacorps propped up by excess liquidity in the market, allowed to hobble along brainlessly and blather about MLK day or pride month on social media to detract from the obvious fact that they’re simulacra of profitable companies. Zombies are the perfect environment for Data Niggers, and less for everyone else.

Why this is so requires some explanation. After all, a very large, gay company may sound enticing to those coming from academia. The salaries are higher, yet it’s historical revisionism to claim there were no rich slaves. The company is more ‘successful,’ but it’s a mistake to conflate this with Capitalism, as destruction of zombies (their existence artificially extended) is as central a component as creation. We have reinvented the planation and called it ‘the Fortune 2000.’ Remember, as I have written about elsewhere, that under postmodernism institutions conceal their types by shedding their superficial signifiers. The Great Leader was always the least characteristic attribute of totalitarian regimes, so a totalitarian regime that strips itself of its leader while its defining mechanisms continue can conceal itself from the chandala. Similarly, we have reinvented the dynamics of the planation without the slavemaster, the man behind the curtain who has been abstracted away.

Zombies are largely those ‘Fortune Cookie Factories’ whose Data Roles are directed toward creating the impression of financial soundness but are in fact exemplary products of the mixed market economy. (You don’t need to be particularly historically knowledgeable to draw parallels to the KPI production in the Soviet Union or Maoist China). Beyond producing digital cotton, there is no meritocracy in a zombie and in their halls you regularly hear thinly-veiled cries of, “Data Nigger, gets yo ass to the KPI fields— nobody’s paying you to read computer science articles!” And it is for this reason these companies are the most flagrant Affirmative Action hiring machines. Diversity is its own metric, its own KPI, that the planation can create through fiat.

Now you can see through the dynamics of self-selection— it’s no coincidence that many A+ students from ‘good universities’ opt into zombies— and afterwards environmental pressure molds individuals into consummate Data Niggers (and niggers of other varieties). Slaves are not lazy in the conventional sense, as often they pick cotton from sunrise to sunset, but all their tasks are well-delineated from above and deviation is punished. We are, in the postmodern era, beyond the traditional ‘rule of thumb’ and on to punishments that are psychological in nature, a Rube Goldberg machine of carrots and sticks, praises and condemnations, petty and womanly competitions. All of this leads to an environment— again, because the Fortune Cookies are not reflective of actual value and accomplishment— that is fundamentally political rather than productive, a struggle among field niggers, house niggers, and the veteran Head Niggers.

It is now cliché to acknowledge that institutions like slavery are perpetuated by the ‘oppressed’ themselves. Female genital mutilation in barbarian tribes and foot binding in historical China were propagated by mothers and aunts— women— who inflicted it on their younger female relatives. You can attribute this to whatever you want, but in the modern world this behavioral pattern is an expression of resentment, the idea that “you need to do this because it’s required to get ahead[, because I did it to get ahead [and I will not see you become successful without succumbing to the same humiliations I had to]].” No Head Nigger would tolerate some former raggedy-ass nigger strolling into his domain, bypassing the humiliation rituals he suffered, and upending carefully cultivated planation routines. And most Head Data Niggers arrived at their present location through the same path: toiling in the fields of Excel spreadsheets, the mass of carefully-colored Powerpoint decks, answering to every whim of their former Head Data Niggers in an endless stream of useless KPI production. You cannot tell a Head Data Nigger that the models do not actually make sense— he’s been treating the permutation importance of a fifty-variable random forest as a causal effect for twenty years and you cannot simply stroll onto the planation and pretend otherwise. Reading is forbidden, you must do as you’re told for 50 hours of meaningless tasks, clock out, and watch Netflix.

And all this leads to an inverted hierarchy. Perhaps it was a historical impossibility that a black slave would have invented the cotton gin, but if it were possible and one tried he probably would have beaten for on-the-clock faggotry. As with all systems where resentment is the organizing psychological principle, natural hierarchies are inverted— ingenuity is punished where groveling and fealty are rewarded. Head Data Niggers excelled at these latter traits while whoever did not moved on to other environments where they could thrive. And, as we learnt from Nietzsche, inverted hierarchies are constituted by resentment toward the natural hierarchies they invert, a distrust of the actual predispositions— intelligence, creativity, risk-taking, competition— that allowed capitalism to flourish. You also shouldn’t confuse mid-level Data Niggers (and others) with the executive level, which often comes into a company from the outside and hangs above the petty politics that characterizes the plantation (the C-suite usually isn’t the slavemaster, which is the company as an abstraction, but they are more representative of that class). Head Data Niggers may become moderately wealthy by accumulating oversized salaries but, trust me, there are more dignified ways to become at least as wealthy.

Figure 1. Head Data Nigger at a Fortune 2000 company screaming because you forgot to replace positive numbers with smiley faces in your powerpoint slides

Concealed Types

Now that you know the company types, it’s a simple matter of applying to the appropriate ones, right? Were it so simple: companies have an incentive to conceal their type. (This overly rationalist framing is not exactly correct, since many people do not know their company’s own type).

When you apply to academic jobs you’re expected to apply narrowly within your own field— a political scientist rarely gets a job in an economics department, or vice versa— which amounts to between 50 and 100 applications a cycle. Each requires superfluous supporting documents: research statement, teaching statement, diversity statement. Invert this when applying to industry jobs. If you’re an ABD, or comfortably tenured, perhaps you will target Facebook, Amazon, Google, and other established tech companies and secure these jobs. This is fine, and often pays off well, but the beauty of the nonacademic market is its anarchy.

Here’s an example from my first job search, where I fired off my resume to hundreds of positions without heeding the ads. One of the first places I interviewed was a mediocre company that was branching out to specialize in a niche area in which I had demonstrable expertise (e.g., I could reasonably be considered one of the few people who published academically on the subject and had piloted a solution to this problem for USG). They rejected me after two rounds of interviews and, I learned later because I interviewed the person two years afterwards, the person they hired had five years of experience in a completely unrelated area. A few weeks later I received an interview request and realized it was from a startup I was seriously unprepared for; everything specified in the job ad was alien to me. During the interviews I realized the 30-person company was mostly STEM PhDs, mostly in engineering and computer science, some from elite universities; they offered me the job and it was the best place I’ve worked since.

In short, if you’re time-pressed or otherwise inclined it makes sense to apply broadly and quickly. Another reason is to allow a job to disclose its type. If you accept the first offer you’re given, you might be surprised that its type is one better than you anticipated. If it’s one worse, then you can keep applying to jobs as you’re performing your main duties and switch roles; no matter how tight the labor market gets, it will always be better than the academic market. Another way to handle uncertainty surrounding a job’s type is to accept multiple jobs simultaneously, if that opportunity arises. You cannot do this as an academic— it’s difficult to simultaneously accept jobs at the University of Arkansas and the University of Alaska— but you can in industry, so long as the positions are hybrid or remote eligible. If one position turns out to be a worse type than anticipated, you have an alternative to rely on and can cut the undesirable one. A couple years back I onboarded into two new jobs in the same week. One I anticipated to be a startup but was already well into its mediocre stage. Here the benefit was that the work, while unexciting, was relatively easy and I could accomplish it to satisfaction in ten to fifteen hours a week. My boss never pried because the company was at a stage where performing simple work to satisfaction was profitable for it (and management was aware its salaries for astrologer roles was not competitive). The other job, on the other hand, had promise as a startup in a zombie but turned out to be a Zombie. It paid considerably more but was dismal. It was possible to scrape by with tedious and not terribly important or well-executed work, which was all that was bureaucratically possible in that organization. I held both positions for exactly ten months, getting the benefit of double salaries (thanks to the larger one provided by the Zombie) while having light yet moderately interesting and business-essential work (thanks to the mediocre) .

With a tentative strategy in mind, let’s return to the sociological question: why do companies conceal their types? Zombies have two options: to represent themselves as mediocre to attract talent by offering light workloads or to represent their positions as prized ‘startups in a big gay company’ to attract ambitious talent. In either case, the person wants to hire you to communicate it as a ‘win’ to upper management and then gradually force you into the Data Nigger occupation. Behind closed doors, your immediate supervisors who hired you by deliberately falsifying their type bellow, “Massa! Massa! Come look what yo Head Data Nigger done. He done got ‘imself a great ol’ big brain PhD nigga! Oh yessir he gon look gud on yus recruitment KPI, yessir.” It doesn’t particularly matter whether you are appropriate for the role and whether it ought to have been given to a recent college grad— I do not mean this condescendingly, since if you fill PhDs into roles best reserved for BAs it makes it harder for the latter to find well-remunerated jobs— because the goal is to fulfill a self-justifying recruitment KPI. Head Data Nigger thinks it’s possible to ease you into niggerdom because you become bound to the job, which is a secondary reason it makes sense to have ample backups available. (Note: I am, if it really needs saying, a bit facetious here. Many a Head Data Nigger is actually a fine, amiable person who has the best intentions. It is the structure of the Zombie that forces this pattern on its workers).

On the other side, a genuine startup in a zombie has incentive to conceal its type due to ambiguity about your type. ‘Startups in a zombie’ have a not totally undeserved negative reputation among zombies, based on the idea they don’t actually produce value. Sometimes this is true. Other times they produce tremendous value but it’s not immediately obvious to the larger organization, which has information gaps between technical teams and non-technical management (in an inverted hierarchy, ‘leadership’ may be incompetent and unwilling to learn math beyond a seventh grade level). To conceal this internally, a good startup in a zombie often pretends to spends its time on KPI maximization when in reality those reports take up perhaps 5 percent of its time. Were the executives to realize this, they would overextend into the mock-startup to redirect the team to Fortune Cookie optimization and inadvertently kill substantial value. The worst outcome is if they hire you as a Data Astrologer but your real type is Data Nigger, because you will blather to upper-level management about things they believe are inconsequential and flood the team with Fortune Cookie requests as the execs realize ten times as many KPIs can be squeezed out. Good supervisors will try to uncover your true type but subtly to not give away the game; that way, if your true type is Data Nigger you can be siloed into KPI production while the astrologers utter incantations, while an astrologer would simply leverage their natural autonomy to produce whatever they can infer would be both interesting and valuable.

Startups are less likely to conceal themselves because it’s more difficult to mistake them for mediocre or zombie companies. More recently I have noticed startups concealing themselves as mediocre companies to attract talent in a more precarious financial environment. Some have stopped offering stock options and overplay their long term viability (which is also less certain for genuine startups). Some zombies, however, can conceal themselves as startups via ‘consulting’. Some small consulting firms that specialize in data work have zombie clients, largely in government but often in industry as well. The fact remains: if your primary client is a zombie you will be increasingly tasked with Data Nigger work, no matter what ‘culture’ the firm you operate from purports to have. Take special care when interviewing with consulting firms to gain as much insight into its primary clients as possible. This ties back to a broader issue, outside today’s purview, of the state concealing its operations with ‘private sector’ firms that operate solely on public money but obscure how much of the nominal private sector is engaged in wholly unproductive labor.

Salary

Everyone wants to know about money. Here’s another trend: salary depends more on company type than role. Within startups, your salary will be determined meritocratically, as in based on your skills and your ability to use them to ensure your company trounces the competition. If you’re a big-brained AI PhD, then you can probably make upwards of half a million dollars annually at the right startup; otherwise, your salary will probably lean closer to low six figures. Mediocre companies tend to have flatter but still meritocratically-based salaries. In the best mediocre company I worked for my salary was 110,000 dollars a year, which tech friends assure me is near poverty. However, I believe the median salary for the company was approximately 75,000 a year as there were many non-technical roles involved. The fact is the company was aware it needed an Astrologer in an area I know well, could compensate the role better than others in the company, but that the pay still wasn’t competitive with other companies in the Astrological realm. While the money wasn’t great, this realization meant it could offer other benefits: I could work exclusively on Astrological tasks I enjoyed, I could actually spend time to do this work well, had basically infinite free time so long as necessary tasks were completed, and I could ignore all bureaucratic minutiae.

Zombies tend to pay the best, unless you’re instrumental at a startup or other tech company, but the pay is tenuously related to ability. Ignoring the bloated HR department, diversity committees, copious AA hires, pedophile directors of safety, the highest paid data positions likely are veteran Head Data Niggers who rose through the ranks of ‘barchart engineering’. More than once I’ve seen Head Data Niggers, who lacked even basic statistics knowledge, confidently micromanage stats PhDs on how to specify their regression models. It doesn’t matter that the Head Nigger doesn’t know the subject, and deliberately wants it done incorrectly, because he’s demonstrated fealty. The would-be astrologer turned Data Nigger complies for the slightly higher salary (compared with a mediocre company or startup, even if their within-company pay is substantially lower than the Head Data Niggers, which is all he can aspire to become after a certain point).

What should you request as salary? If you need a job quickly, go ahead and accept whatever first offer you’re given. After that point, I wouldn’t accept an offer lower than 90,000 and for that price you should always continue your application process in case the company’s revealed type is other than expected (that way you can transfer quickly or, if the lower-paid job is otherwise ideal you can accrue some additional income). Generally, your target should be between 120 and 150k with at least one to two years experience, adjusted upwards if the job demands working in a high COL city (I work remotely from the beautiful and cheap Texas coast, so I can afford to work at a discount— my highest paying job has been 165k with a few years experience). You should adjust upwards if you’re drawn unwittingly, or out of desperation, into a zombie. Not only is this compensation for the psychological torment, but also due to time colonization— a zombie will inundate you with trivial meetings, another distraction from its triviality, leaving little time to even apply elsewhere and enslave you into data niggerdom.

Remember a 160k salary is approximately the 90th percentile and 75th percentile income of political science and economics professors, respectively. You should take advantage not only of the salary but the ability to live wherever you want. With some ‘lifestyle engineering’, you can easily live better than basically all professors in your field, the counterfactual for what you would have become had you not moved to industry, save for the Nobel prize winners. It does take some cunning to pull this off while not sacrificing your freedom, which you can increase relative to academia if you’re cunning about it. Chances are you didn’t consider this route only to become a slightly better paid non-academic Data Nigger.

Part II. On the knowledge you need if you were retarded in grad school, coming eventually