ioctl kernel call failing - ioctl module is in kernel, puppy/ubuntu Linux VM in Oracle's Virtual Box

[incognitodiscforums]

Hi, I was wondering if anyone on the Linux forum might have any advice for understanding what is happening here.

This question is in regard to my ioctl error call from the user space to kernel space in a Ubuntu Linux OS running as a Virtual Machine in Oracle's Virtual Box. After the module code was compiled, it was loaded into the Linux kernel using the "sudo insmod ioctl_module.ko" command and you can see that succeeded by the message listing below. For some reason, the ioctl call from the user space to the kernel module function says one of the arguments is invalid, and I don't see that it is. (Also, this is only the 2nd time I've ever posted a question to stack overflow, so I hope I followed your conventions correctly!) Thanks.

user level code error message:

saasbook@saasbook: ./ioctl_test

size of testStruct.field1 = 1

size of testStruct.field2 = 1

first ioctl: Invalid argument

code listing for ioctl_test.c:

https://gist.github.com/KarenWest/6629582

can see ioctl_module is in kernel:

/invokeKernelServices$ dmesg

[13563.849593] Loading module

[13578.846063] pseudo_device_ioctl

code_listing for ioctl_module.c:

https://gist.github.com/KarenWest/6629557

Here are some more specifics on the problem. Using the ioctl.h bit definitions, using the #define macros for _IOW, shifting and oring them together, I see that the IOCTL_TEST = 0x4002006 is correct, and the cmd = 0x804a030 is incorrect, but I cannot figure out why cmd has the wrong value. cmd is an argument to the above ioctl entry point:

static int pseudo_device_ioctl(struct inode *inode, struct file *file,
unsigned int cmd, unsigned long arg)

So I had a look at the proc_fs.h input file's proc_dir_entry structure, where the proc_fops pointer in that structure is set to my ioctl entry point function, with pseudo_dev_proc_operations initialized to a file_operations structure, and proc_entry initialized as a pointer to a proc_dir_entry structure (all in code links on git above), with all these initialized in the ioctl_module.c's initialization_routine(), which I understood had set up this ioctl with the correct values, once code running at the ioctl_test.c level calls, and the OS should be set up to direct it to my ioctl function, and it does. However, the argument sent to my ioctl function set up (cmd) is incorrect, and does not match IOCTL_TEST, which is why this fails.

pseudo_dev_proc_operations.unlocked_ioctl = pseudo_device_ioctl;
pseudo_dev_proc_operations.compat_ioctl = pseudo_device_ioctl;

proc_entry = create_proc_entry("ioctl_test", 0666, NULL);
proc_entry->proc_fops = &pseudo_dev_proc_operations;

So I followed the proc_fs.h's proc_fops pointer (which is initialized to pseudo_dev_proc_operations, which takes it to my ioctl pseudo_device_ioctl function) to see why the cmd argument might get messed up in processing, and I am lost - no idea why this is happening.

If anyone has any further comments they can make that may help me figure out where I've gone wrong, please let me know. Thanks!

 

[ryanvade]

Why do you need ioctl in a VM?
http://man7.org/linux/man-pages/man2/ioctl.2.html

 

[incognitodiscforums]

Why do you need ioctl in a VM?
http://man7.org/linux/man-pages/man2/ioctl.2.html

The idea is to understand how modules are inserted into the Linux kernal running as a VM on (in this case) Oracle's Virtual Box. Once the module (my ioctl_module.c) is loaded into the kernel, if you compile a piece of code at the user level (my ioctl_test.c), then you set up an ioctl to invoke the kernel services you just loaded into the kernel via the module. It's an exercise in understanding how to communicate between the user level and the kernel, with this part of the kernel being a module you also loaded. That's all this is - and the kernel happens to be a VM within the Oracle Virtual Box, in this case, I have both Puppy Linux running and Ubuntu. So it may not be something that is actually realistic that you would do in a VM, but rather to teach you how this works.

So now that I explained that, do you have any idea what may be going wrong here? The cmd argument to my ioctl function should be equal to the IOCTL_TEST set up by _IOW (ioctl.h).
Thanks.

 

[incognitodiscforums]

This might help - just read this:

http://tldp.org/LDP/lkmpg/2.6/html/x892.html