Trouble Calling ioctl from user-space C

Question

I'm trying to implement a program to access memory on an embedded system. I need to access some control register so I think that ioctl is the best way to do it. I have added the ioctl to the fops:

struct file_operations aes_fops = {
  read: aes_read,
  write: aes_write,
  unlocked_ioctl: aes_ioctl,
  open: aes_open,
  release: aes_release
};

And have set up the function:

int aes_ioctl(struct inode *inode,  
     struct file *file, 
     unsigned int ioctl_num,    
     unsigned long ioctl_param){

     printk(KERN_INFO "in ioctl\n");
....
}

But I am not getting inside of this function. Here is my user space code. Please help me understand if I am doing this totally wrong.

int main(int argc, char* argv[]){
    int fd = fopen("/dev/aes", "r+");
    ioctl(fd, 0, 1);
    fclose(fd);
}

Some of the code is apparently for older kernels, because I am compiling for an embedded system where an older version of Linux has been modified.

Answer 1

Without knowing the error returned, it's hard to say... My first though is your permissions on your file descriptor. I've seen similar issues before. First, you can get some more information about the failure if you take a look at the return of the ioctl:

#include <errno.h>
int main(int argc, char* argv[])
{
    long ret;     
    int fd = fopen("/dev/aes", "r+");
    ret = ioctl(fd, 0, 1);
    if (ret < 0)
        printf("ioctl failed. Return code: %d, meaning: %s\n", ret, strerror(errno));
    fclose(fd); 
} 

Check the return values and this should help give you something to search on. Why check? See the bottom of the post... Next in order to check if it is permissions issue, run the command:

ls -l /dev/aes

if you see something like:

crw------- 1 root root 10, 57 Aug 21 10:24 /dev/aes

Then just issue a:

sudo chmod 777 /dev/aes

And give it a try again. I bet it will work for you. (Note I ran that with root permissions since root is the owner of my version of your mod)

If the permissions are already OK, then I have a few more suggestions:

1) To me, the use of fopen/fclose is strange. You really only need to do:

int fd = open("/dev/aes");
close(fd);

My system doesn't even let your code compile as is.

2) Your IOCTL parameter list is old, I don't know what kernel version your compiling on, but recent kernels use this format:

long aes_ioctl(struct file *file, unsigned int ioctl_num, unsigned long ioctl_param){   

Note the removal of the inode and the change of the return type. When I ran your code on my system, I made these changes.

Best of luck!

Note: Why should we check the return when we're "not getting into the ioctl"? Let me give you an example:

//Kernel Code:
//assume include files, other fops, exit, miscdev struct, etc. are present

long hello_ioctl(struct file *file, unsigned long ioctl_num, unsigned long ioctl_param) {
    long ret = 0;
    printk("in ioctl");
    return ret;
}

static const struct file_operations hello_fops = {
    owner: THIS_MODULE,
    read: hello_read,
    unlocked_ioctl: hello_ioctl,
};

static int __init hello_init(void) {
    int ret;
    printk("hello!\n");
    ret = misc_register(&hello_dev); //assume it worked...
    return ret;
}

User space code:

//assume includes

void main() {
  int fd;
  long ret;
  fd = open("/dev/hello");
  if(fd) {
    c = ioctl(fd, 0, 1);
    if (c < 0)
      printf("error: %d, errno: %d, meaning: %s\n", c, errno, strerror(errno));
    close(fd);
  }
  return;
}

So what's the output? Lets assume bad file permissions on /dev/hello (meaning our user space program can't access /dev/hello).

The dmesg | tail shows:

[ 2388.051660] Hello!

So it looks like we didn't "get in to" the ioctl. What's the output from the program?

error: -1, errno: 9, meaning: Bad file descriptor

Lots of useful output. Clearly the ioctl call did something, just not what we wanted. Now changing the permissions and re-running we can see the new dmesg:

[ 2388.051660] Hello!
[ 2625.025339] in ioctl

Answer 2

The problem with your code is the request number you are using - 0. The kernel has some request number reserved for internal use. The kernel regards the request number as a struct, separates it to fields and calls the right subsystem for it.

See Documentation/ioctl/ioctl-number.txt (from Linux 3.4.6):

Code  Seq#(hex) Include File            Comments
========================================================
0x00    00-1F   linux/fs.h              conflict!
0x00    00-1F   scsi/scsi_ioctl.h       conflict!
0x00    00-1F   linux/fb.h              conflict!
0x00    00-1F   linux/wavefront.h       conflict!
0x02    all     linux/fd.h
0x03    all     linux/hdreg.h
...

Depending on what you are during, you'd have to follow the kernel guidelines for adding new ioctls():

If you are adding new ioctl's to the kernel, you should use the _IO
macros defined in <linux/ioctl.h>:

    _IO    an ioctl with no parameters
    _IOW   an ioctl with write parameters (copy_from_user)
    _IOR   an ioctl with read parameters  (copy_to_user)
    _IOWR  an ioctl with both write and read parameters.

See the kernel's own Documentation/ioctl/ioctl-decoding.txt document for further details on how those numbers are structured.

In practice, if you pick Code 1, which means starting from 0x100 up until 0x1ff, you'd be fine.

Answer 3

Is your setup of the aes_fops structure correct? I've never seen it done that way. All the code I have is:

.unlocked_ioctl = aes_ioctl,

rather than:

unlocked_ioctl: aes_ioctl,

Colons within a structure (as you have in your setup) fields are used for bit fields as far as I'm aware (and during definition), not initialising the individual fields.

In other words, try:

struct file_operations aes_fops = {
    .read            = aes_read,
    .write           = aes_write,
    .unlocked_ioctl  = aes_ioctl,
    .open            = aes_open,
    .release         = aes_release
};

---

Note: It appears that gcc once did allow that variant of structure field initialisation but it's been obsolete since gcc 2.5 (see here, straight from the GCC documentation). You really should be using the proper method (ie, the one blessed by the ISO standard) to do this.