Can I programmatically log into a website without storing the password in plaintext?

Question

I do a number of projects that involve automatically submitting forms and/or retrieving data from websites. Some of these sites require username/password authentication. (These sites do not have APIs, so I am relying on screen scraping.)

Most of the tutorials I've seen store the username and password in the source code like any other POST data, e.g.:

string username = "someUserName";
string password = "somePassword";
// submit POST data...

But I know storing passwords in plain text is generally frowned upon. Is there an alternative method I should use?

Answer 1

I have a scraping project that needed this problem solved. My setup includes two separate servers. The first is the user front end web app. the second is a nodejs server that handles the scraping.

I handle encryption with openssl key pair encryption. I generate a key pair for the nodejs machine and give the public key to the front end web app. When a user registers their 3rd party credentials those credentials are encrypted with the public key and stored in a database.

The web app regularly selects a user's encrypted credentials and sends them to the node server where they are decrypted with the private key and used with the 3rd party for scraping.

After a quick search I found this article about using openssl and encrypting strings.

I realize this is a very old post but hopefully it helps the next person that stumbles onto this problem.

Answer 2

There's no way to do it. it will need to be available to the script somewhere as plain text (or "reversible encryption").

Many Apis (including Amazon Web Services for example) will recommend setting credentials in a environment variable and this is probably as much safety as you can hope for.

Put it in your .bash_profile, double check perrmissions, and at least you can be sure it won't end up on github in a public repo.

Answer 3

A pattern we use is:

In your database table you have an encrypted column. This column contains data encrypted with a system-wide, long (128 bit), random secret key (usually stored in a configuration file). The data in this encrypted column contains a separate (random) secret key used for each thirdparty service. With this password we encrypt the authentication details related to this thirdparty service.

Why this double encrypting?

You reduce the amount of passwords in plain text to a single one (the system-wide password). Because of this, key management is easier. We create a long random secret key for each thirdparty service so that we can selectively decrypt the credentials for each thirdparty service and transfer them between systems if necessary. Having one of our secret keys stored outside the database also reduces the risk associated with both SQL-injection attacks (they 'only' get the database data) and with backups (configuration files are not included in the regular backup data).

The weakness is obviously the system-wide password. It needs to be in memory somewhere.

I'm no cryptographer, and I'm pretty sure the above is sub-optimal. However, it works, is manageable and lot safer than just storing the thirdparty service credentials in plain text.

Answer 4

You have to do two things:
1. Use HTTPS for login pages (if necessary)
2. Use password Encryption right after receiving it. An encoder is something like this:

private static String passwordEncryption(String oldPass){
    String newPass = "";
    try {
        MessageDigest messageDigest = MessageDigest.getInstance("MD5");  
        messageDigest.update(oldPass.getBytes(), 0, oldPass.length());  
        newPass = new BigInteger(1,messageDigest.digest()).toString(16);  
        if (newPass.length() < 32) {
            newPass = "0" + newPass; 
        }
        return newPass;
    } catch (NoSuchAlgorithmException e) {
        e.printStackTrace();
    }
    return newPass;

}

And use MD5() function of MySql to compare the received password with the stored one.

Answer 5

A very simple way to encrypt and decrypt is extended tiny encription algorithm (XTEA). I'm pasting the C++ code from wikipedia here, but keep in mind anyone could have changed it there.

#include <stdint.h>

/* take 64 bits of data in v[0] and v[1] and 128 bits of key[0] - key[3] */

void encipher(unsigned int num_rounds, uint32_t v[2], uint32_t const key[4]) {
    unsigned int i;
    uint32_t v0=v[0], v1=v[1], sum=0, delta=0x9E3779B9;
    for (i=0; i < num_rounds; i++) {
        v0 += (((v1 << 4) ^ (v1 >> 5)) + v1) ^ (sum + key[sum & 3]);
        sum += delta;
        v1 += (((v0 << 4) ^ (v0 >> 5)) + v0) ^ (sum + key[(sum>>11) & 3]);
    }
    v[0]=v0; v[1]=v1;
}

void decipher(unsigned int num_rounds, uint32_t v[2], uint32_t const key[4]) {
    unsigned int i;
    uint32_t v0=v[0], v1=v[1], delta=0x9E3779B9, sum=delta*num_rounds;
    for (i=0; i < num_rounds; i++) {
        v1 -= (((v0 << 4) ^ (v0 >> 5)) + v0) ^ (sum + key[(sum>>11) & 3]);
        sum -= delta;
        v0 -= (((v1 << 4) ^ (v1 >> 5)) + v1) ^ (sum + key[sum & 3]);
    }
    v[0]=v0; v[1]=v1;
}

Answer 6

The common way of storing a password is by hashing it. As most algorithms for hashing passwords are destructive, that is they can't be reversed, this wouldn't work for you.

An option would be to use a reversible hash, such as to base64 encode the password, but it isn't really a lot safer than storing it in plain text.

The best solution as far as I can see, would be to store the passwords in a database. If you are really worried about someone getting the usernames and passwords, you could encrypt them in the DB with encryption functions, or you could use a SQLite database which you would encrypt directly on the disk.

This way your code and login credentials are separated, and you can safely share your code with others without worrying about security.