24 Data Science Articles from May 2022

/

Spark or Hadoop? Both Apache products can be used by data scientists but which is the better analytics tool? Here’s a comparison—along with which one will fit better based on your project focus

Fixing data lake errors can be time-consuming, and costly—here are some thoughts on standardized an autonomous validation approach to avoid the lake becoming a swamp

Interpol: in a couple of years expect state-developed cyber weapons to be available on the dark net  

Do you think Python is slow? Here’s a fast way to loop in Python

Looking for patterns in satellite image time series with python? Here’s a quick guide for handling time-varying imagery with open python libraries 

Can the new-and-improved Large Hadron Collider save particle physics?   

Want to run Python code in a browser? Soon you might be able to 

The AI Engineering Process: A guide to solving an AI problem

The challenges of organizing geospatial intelligence efficiently 

Making predictions outperforms smart teams of data scientists working on large data sets. Some examples of machine learning mistakes thanks to the narrow thinking of the humans that created them.

Some researchers claim we’re on the cusp of GoPro physics—where a camera can point at an event and an algorithm can identify the underlying physics equation

An in-depth look at Neural architecture search—the AutoML subfield aiming to replace manual designs

In an effort to enhance artificial intelligence & machine learning technologies military researchers are letting it be known they want more accurate processing of covariance information related to environmental variations and noise

Intelligence agencies are starting to coalesce around a set of common standards and data for using open source intelligence

A detailed explanation of handling satellite imagery in the format of .tiff files using Python.

A way to better understand road networks by measuring their spatial homogeneity using machine-learning tools like graph neural networks

The place where machine learning shines

A new deep learning technique shows promise to make robotics systems more stable in handling deformable objects

Small satellites: The implications for national security 

Ukraine may be a tipping point for developing intelligent weapons

Two main types of adversarial attacks in neural networks

It’s not just about gathering data—it’s telling compelling stories 

NGA to Leverage AI, ML for GEOINT Analysis at Scale 

From data scientist to … comedian?

Follow for a daily Data Science article

Plan to Adapt

/

The primary message of (many career) books and countless others is to listen to your heart and follow your passion. Find your true north by filling out worksheets or engaging in deep, thoughtful introspection. Once you’ve got a mission in mind, these books urge, you’re supposed to develop a long-term plan for fulfilling it. You’re supposed to craft detailed, specific goals. You’re urged to figure out who you are and where you want to be in ten years, and then work backward to develop a roadmap for getting there.  

This philosophy has some serious strengths. It’s important to have worthy aspirations. If you are passionate about something, you’ll have fun, stay committed, and achieve more. It’s also right to invest for the long term: to find out whether you’re good at something and whether you like it, you need to stick with it for a meaningful amount of time.  

But it presumes a static world. You will change. The environment around you will change. Your allies and competitors will change. It’s unwise, no matter your stage of life, to try to pinpoint a single dream around which your existence revolves.  

Reid Hoffman and Ben Casnocha from The Startup of You

What you'll be like a decade from now

/

Why do people get ill-advised tattoos, marry questionable partners, or make financial-planning decisions they come to regret? A new study suggests that part of the reason is that we aren’t very good at predicting how much we’re going to change in the future. We are prone to believe whatever we think and value now will hold true. Psychologist Daniel Gilbert led the study and says, “People really aren’t very good at knowing who they’re going to be and hence what they’re going to want a decade from now.” Gilbert tells LiveScience.com, “At every age we think we’re having the last laugh, and at every age we’re wrong.”

The Harvard University study survey of more than 19,000 people between the ages of 18 and 68. People act as if history shaped them and then ended, leaving them in their final form. The researchers call the effect “the end of history” illusion.

Younger people in the survey did not expect to change as much as their the elders changed within the same time frame. The researchers made an effort to make sure that the people in the survey were not just overestimating past change but rather underestimating future change by comparing the results to predictions made on another survey a decade ago.

Although we aren’t very good at predicting our future selves, most of us are able to see that our values, preferences and personalities are different from a decade ago. We just can’t predict how much change will come looking forward the same length of time.

We may be motivated by the desire to comfort ourselves. We tell ourselves that future change won’t be very dramatic. We know ourselves and the future is predictable. Our present selves are permanent, so this thinking goes.

Other studies show you are less likely to change the older you get, but you will still change more than you expect.

Gilbert offers this advice: Take care when making long-term decisions to include a “margin for escape”. If you are buying a ticket to see your favorite band in ten years, you might want to pause before buying a ticket.

But there is another side of the coin to consider before including a 10 year opt-out clause in your wedding vows: Research shows that when people feel they have the ability to change their minds, they're less happy with the choices they've made.

You can read more about the study in the journal Science.

Stephen Goforth

15 Career Advice Articles

/

Frankl’s Decision: Meaning or Happiness?

/

By 1941, Viktor Frankl’s theories had received international attention and he was working as the chief of neurology at Vienna's Rothschild Hospital, where he risked his life and career by making false diagnoses of mentally ill patients so that they would not, per Nazi orders, be euthanized.

That was the same year when he had a decision to make, a decision that would change his life. With his career on the rise and the threat of the Nazis looming over him, Frankl had applied for a visa to America, which he was granted in 1941. By then, the Nazis had already started rounding up the Jews and taking them away to concentration camps, focusing on the elderly first. Frankl knew that it would only be time before the Nazis came to take his parents away. He also knew that once they did, he had a responsibility to be there with his parents to help them through the trauma of adjusting to camp life. On the other hand, as a newly married man with his visa in hand, he was tempted to leave for America and flee to safety, where he could distinguish himself even further in his field.

As Anna S. Redsand recounts in her biography of Frankl, he was at a loss for what to do, so he set out for St. Stephan's Cathedral in Vienna to clear his head. Listening to the organ music, he repeatedly asked himself, "Should I leave my parents behind?... Should I say goodbye and leave them to their fate?" Where did his responsibility lie? He was looking for a "hint from heaven."

When he returned home, he found it. A piece of marble was lying on the table. His father explained that it was from the rubble of one of the nearby synagogues that the Nazis had destroyed. The marble contained the fragment of one of the Ten Commandments -- the one about honoring your father and your mother. With that, Frankl decided to stay in Vienna and forgo whatever opportunities for safety and career advancement awaited him in the United States. He decided to put aside his individual pursuits to serve his family and, later, other inmates in the camps.

The wisdom that Frankl derived from his experiences there, in the middle of unimaginable human suffering, is just as relevant now as it was then: “Being human always points, and is directed, to something or someone, other than oneself — be it a meaning to fulfill or another human being to encounter. The more one forgets himself — by giving himself to a cause to serve or another person to love — the more human he is.” 

Emily Esfahani Smith writing in The Atlantic

Murder Confession Tattoo

/

A Los Angeles gang member was convicted of murder because he had his crime tattooed on his chest. Seven years after the 2004 killing, the 25-year-old was arrested for another crime. That’s when a police officer noticed the scene inscribed on the man’s torso. It included a lifeless body and the outline of a liquor store. A law enforcement officer went into the man’s cell, pretending to be another member of the same gang. The confession given to the undercover cop led to the man’s conviction on first-degree murder charges.

Each of us has secrets hidden underneath carefully guarded masks. Whether or not we wear them as tattoos, their itch reminds us of how they have impacted our lives.

Stephen Goforth

Harmful comments at Work

/

Families and workplaces have a tendency to overlook people’s harmful comments or behavior out of respect for a shared history. At family gatherings, “sometimes there’s this idea of like, Oh, well, that’s just Uncle Larry. He might say some homophobic or racist, sexist stuff, but that’s just how he is,” Cynthia Pong, a New York City–based career coach said. “I have seen that happen before in the workplace, excusing people who’ve maybe been around the company for some time, and really not holding them to account for the highly problematic things that they may be saying or doing.”

Joe Pinsker, writing in The Atlantic

Upcoming Webinars: Mental Health in Media, Disinformation

/

Wed., May 25 - Mental health in media

What: A discussion of mental health portrayals on-screen.

Who: Various industry leaders.

When: 10 am, Pacific

Where: Zoom

Cost: Free

Sponsor: USC Annenberg School of Journalism

More info

Sat., June 5 - How to Cover Race Workshop

THIS EVENT IS NO LONGER SCHEDULED

Fri., June 10 - Facts in a Time of Fiction: Reporting the truth amid lies and disinformation

What: How misinformation and lies spread after the 2012 Sandy Hook school shooting, along with insights on how conspiracy theories grow.

Who: New York Times writer and author Elizabeth Williamson, whose critically-acclaimed book “Sandy Hook: An American Tragedy and the Battle for Truth” offered on-the-ground reporting to trace a line from conspiracy theories around Sandy Hook to Jan. 6, 2021.

When: 11:30 am, Eastern

Where: Zoom

Cost: Free

Sponsor: National Press Club Journalism Institute

More info

12 apps for Job hunters

/

Career Builder - online hiring app that allows job seekers to access tools that will help them at every point in the process.

ExpressJob - mapping that shows nearby jobs and makes applying easy with one-click applications but also offers ways to stay organized once you are hired (timesheets, schedule, etc.)

Glassdoor - search engine platform offering job openings along with company reviews.

Good & Co. - Uses Myers-Briggs to help users know whether a job will be good fit.

Hirect - chat-based-direct hiring platform.

Hirewire - rather than upload a resume, build an interactive profile for employers to check out. Mostly service industry positions.

Indeed - sort through the search engine database and stay on top of openings that interest you. 

Linkedin - the social network for professionals.

Linkup - focuses on little-known job listings. Free, iOS only. 

Monster - brings jobs from other job searchers into a single app.

Snagjob - only hourly jobs. Free.

ZipRecruiter - offers more than 100 job boards with filters. Sends notifications about vacancies.

More job hunting help

Emotional relief is not the same as emotional recovery

/

Venting can be like scratching a mosquito bite. It feels like it works at first. Studies have shown a drop in diastolic blood pressure of 1 to 10 points after venting. But they show no attendant drop in hostility. It feels like we release anger or frustration, but we don’t. Even if we didn’t experience this temporary alleviation, there’s the fact that negative feelings naturally dissipate over time. People who do nothing assume the abatement owes to time; people who vent believe venting did the trick. And our choices can be self-reinforcing. If it seems like venting worked, we’re less likely to abide by social norms around holding back in the future. 

Gail Cornwall & Juli Fraga writing in Slate

Commercial Culture’s Substitute for Love

/

Jonathan Franzen gave this commencement speech  on May 21, 2011 at Kenyon College

Let me toss out the idea that, as our markets discover and respond to what consumers most want, our technology has become extremely adept at creating products that correspond to our fantasy ideal of an erotic relationship, in which the beloved object asks for nothing and gives everything, instantly, and makes us feel all powerful, and doesn’t throw terrible scenes when it’s replaced by an even sexier object and is consigned to a drawer. 

To speak more generally, the ultimate goal of technology, the telos of techne, is to replace a natural world that’s indifferent to our wishes — a world of hurricanes and hardships and breakable hearts, a world of resistance — with a world so responsive to our wishes as to be, effectively, a mere extension of the self. 

Let me suggest, finally, that the world of techno-consumerism is therefore troubled by real love, and that it has no choice but to trouble love in turn. 

Its first line of defense is to commodify its enemy. You can all supply your own favorite, most nauseating examples of the commodification of love. Mine include the wedding industry, TV ads that feature cute young children or the giving of automobiles as Christmas presents, and the particularly grotesque equation of diamond jewelry with everlasting devotion. The message, in each case, is that if you love somebody you should buy stuff. 

A related phenomenon is the transformation, courtesy of Facebook, of the verb “to like” from a state of mind to an action that you perform with your computer mouse, from a feeling to an assertion of consumer choice. And liking, in general, is commercial culture’s substitute for loving.  

The striking thing about all consumer products — and none more so than electronic devices and applications — is that they’re designed to be immensely likable. This is, in fact, the definition of a consumer product, in contrast to the product that is simply itself and whose makers aren’t fixated on your liking it. (I’m thinking here of jet engines, laboratory equipment, serious art and literature.) 

But if you consider this in human terms, and you imagine a person defined by a desperation to be liked, what do you see? You see a person without integrity, without a center. In more pathological cases, you see a narcissist — a person who can’t tolerate the tarnishing of his or her self-image that not being liked represents, and who therefore either withdraws from human contact or goes to extreme, integrity-sacrificing lengths to be likable. 

If you dedicate your existence to being likable, however, and if you adopt whatever cool persona is necessary to make it happen, it suggests that you’ve despaired of being loved for who you really are. And if you succeed in manipulating other people into liking you, it will be hard not to feel, at some level, contempt for those people, because they’ve fallen for your shtick.  

Consumer technology products would never do anything this unattractive, because they aren’t people. They are, however, great allies and enablers of narcissism. Alongside their built-in eagerness to be liked is a built-in eagerness to reflect well on us. Our lives look a lot more interesting when they’re filtered through the sexy Facebook interface. We star in our own movies, we photograph ourselves incessantly, we click the mouse and a machine confirms our sense of mastery. 

And, since our technology is really just an extension of ourselves, we don’t have to have contempt for its manipulability in the way we might with actual people. It’s all one big endless loop. We like the mirror and the mirror likes us. To friend a person is merely to include the person in our private hall of flattering mirrors. 

I may be overstating the case, a little bit. Very probably, you’re sick to death of hearing social media disrespected by cranky 51-year-olds. My aim here is mainly to set up a contrast between the narcissistic tendencies of technology and the problem of actual love. My friend Alice Sebold likes to talk about “getting down in the pit and loving somebody.” She has in mind the dirt that love inevitably splatters on the mirror of our self-regard. 

The simple fact of the matter is that trying to be perfectly likable is incompatible with loving relationships. Sooner or later, for example, you’re going to find yourself in a hideous, screaming fight, and you’ll hear coming out of your mouth things that you yourself don’t like at all, things that shatter your self-image as a fair, kind, cool, attractive, in-control, funny, likable person. Something realer than likability has come out in you, and suddenly you’re having an actual life. 

Suddenly there’s a real choice to be made, not a fake consumer choice between a BlackBerry and an iPhone, but a question: Do I love this person? And, for the other person, does this person love me?  

There is no such thing as a person whose real self you like every particle of. This is why a world of liking is ultimately a lie. But there is such a thing as a person whose real self you love every particle of. And this is why love is such an existential threat to the techno-consumerist order: it exposes the lie. 

This is not to say that love is only about fighting. Love is about bottomless empathy, born out of the heart’s revelation that another person is every bit as real as you are. And this is why love, as I understand it, is always specific. Trying to love all of humanity may be a worthy endeavor, but, in a funny way, it keeps the focus on the self, on the self’s own moral or spiritual well-being. Whereas, to love a specific person, and to identify with his or her struggles and joys as if they were your own, you have to surrender some of your self. 

The big risk here, of course, is rejection. We can all handle being disliked now and then, because there’s such an infinitely big pool of potential likers. But to expose your whole self, not just the likable surface, and to have it rejected, can be catastrophically painful. The prospect of pain generally, the pain of loss, of breakup, of death, is what makes it so tempting to avoid love and stay safely in the world of liking. 

And yet pain hurts but it doesn’t kill. When you consider the alternative — an anesthetized dream of self-sufficiency, abetted by technology — pain emerges as the natural product and natural indicator of being alive in a resistant world. To go through a life painlessly is to have not lived. Even just to say to yourself, “Oh, I’ll get to that love and pain stuff later, maybe in my 30s” is to consign yourself to 10 years of merely taking up space on the planet and burning up its resources. Of being (and I mean this in the most damning sense of the word) a consumer. 

The fundamental fact about all of us is that we’re alive for a while but will die before long. This fact is the real root cause of all our anger and pain and despair. And you can either run from this fact or, by way of love, you can embrace it. 

When you stay in your room and rage or sneer or shrug your shoulders, as I did for many years, the world and its problems are impossibly daunting. But when you go out and put yourself in real relation to real people, or even just real animals, there’s a very real danger that you might love some of them. 

And who knows what might happen to you then? 

Jonathan Franzen 

Kenyon College, May 21, 2011

 

Wallowing in Guilt

/

No one heaps guilt on themselves when they don't have to do so. Right? Actually, there is much to gain from wallowing in false guilt.

* You gain control over others. You are telling others to orient themselves around your need for 'restoration.'

* Removes responsibility from your shoulders. They can’t expect much from you when you are broken and bleeding, can they?

* You get attention. "Look at me! Don't pay attention to others. Focus on me! See how guilty I am?"

* Get others to pump you up. "I'm so bad." "Oh no, you're a wonderful person.."

* You do for yourself what you wish others would do to you. "I wish other people would punish me because I don't feel worthy of good things happening to me."

* You avoid resentment. It’s easier to admit guilt rather than resentment. What you’d like to do to someone else (punish them) you do to yourself.

* You may have an inadequate view of God. To hang on to your guilt, you distort the truth of forgiveness.

* You may have false guilt because of expectations. You could be trying to live up to everyone’s expectations-and, of course, you can’t. Or you might be trying to live up to a peer group’s expectations.

* You may have a fear of repeating a wrong or the fear of being tempted toward it.

* You may be following some unhealthy teaching. You may have been taught to feel bad over wrongs you’ve committed—forever.

* Your family has been inconsistent in forgiving you and only showed you conditional love.

* You are trying to fulfill unhelpful injunctions you heard from people in childhood. Others may have tried to motivate you by guilt by getting you to focus on the negative aspects of yourself.

* You could be covering up feelings of anger with feelings of guilt. Perhaps you are not making a distinction between guilt and disappointment.

* And finally, you may have done something wrong and are experiencing true guilt but want to avoid dealing with it.

Stephen Goforth

Wanna be creative? Set aside time to do nothing!

/

Doing nothing is creative work. Because when you’re consciously doing nothing, the conscious part is only a tiny part of what your brain is. The rest of it, the unconscious, is chugging away all the time. There’s this process cognitive psychologists call “incubation” – the brain churning over associations. And these associations can pop into awareness as insight. The incubation process is supercharged during sleep, and also when doing nothing, letting your mind wander and having no particular task to perform. 

 If you keep people’s minds busy all the time with tasks, that inhibits this incubation process. I don’t want to say that people should become Luddites and get rid of all the gadgets and become hermits – all that provides raw data for incubation. 

But what we need is a balance between doing nothing and doing something – we need both to fuel creativity and insight.

Brigid Schulte writing in the Washington Post