Tuesday, April 10, 2012

Some tips on Android suspend/resume!

Introduction to suspend
Suspend have 3 major part: Freezing process and tasks Call every driver's suspend callback Suspend CPU and core system devices Freezing process is like stop all process, and when resume, it will start execute as if not stop ever. User space process and kernel space taskes will never know this stop, They are like babies at all. How user let Linux goto suspend ? User can read/write sys fs file: /sys/power/state to control and get kernel power managment(PM) service. such as:
# echo standby > /sys/power/state
to let system going to suspend. also
# cat /sys/power/state
to get how many PM method you kernel supported.
Normal Linux Suspend
Files:
you can checkout a standard linux source code, below is the path.
linux_soruce/kernel/power/main.c linux_source/kernel/arch/xxx/mach-xxx/pm.c
Let 's going to see how these happens. The userspace interface /sys/power/state is state_store() function in main.c: You can write the strings defined by const char * const pm_state[]: such as "mem", "standby". In a normal linux kernel, It will going to enter_state() in main.c enter_state() will first do some check of state. sync file system. Below is the source code:
/**
* enter_state - Do common work of entering low-power state.
* @state: pm_state structure for state we're entering.
*
* Make sure we're the only ones trying to enter a sleep state. Fail
* if someone has beat us to it, since we don't want anything weird to
* happen when we wake up.
* Then, do the setup for suspend, enter the state, and cleaup (after
* we've woken up).
*/
static int enter_state(suspend_state_t state)
{
int error;

if (!valid_state(state))
return -ENODEV;

if (!mutex_trylock(&pm_mutex))
return -EBUSY;

printk(KERN_INFO "PM: Syncing filesystems ... ");
sys_sync();
printk("done.\n");

pr_debug("PM: Preparing system for %s sleep\n", pm_states[state]);
error = suspend_prepare();
if (error)
goto Unlock;

if (suspend_test(TEST_FREEZER))
goto Finish;

pr_debug("PM: Entering %s sleep\n", pm_states[state]);
error = suspend_devices_and_enter(state);

Finish:
pr_debug("PM: Finishing wakeup.\n");
suspend_finish();
Unlock:
mutex_unlock(&pm_mutex);
return error;
}
Prepare, Freezing Process
Going to suspend_prepare(), this func will alloc a console for suspend, running suspend notifiers, disable user mode helper, and call suspend_freeze_processes() freeze all process, it will make all process save current state, in the freeze stage, maybe some task/user space process will refuze to going to freezing,it will abort and unfreezing all precess.
/**
* suspend_prepare - Do prep work before entering low-power state.
*
* This is common code that is called for each state that we're entering.
* Run suspend notifiers, allocate a console and stop all processes.
*/
static int suspend_prepare(void)
{
int error;
unsigned int free_pages;

if (!suspend_ops || !suspend_ops->enter)
return -EPERM;

pm_prepare_console();

error = pm_notifier_call_chain(PM_SUSPEND_PREPARE);
if (error)
goto Finish;

error = usermodehelper_disable();
if (error)
goto Finish;

if (suspend_freeze_processes()) {
error = -EAGAIN;
goto Thaw;
}

free_pages = global_page_state(NR_FREE_PAGES);
if (free_pages < FREE_PAGE_NUMBER) {
pr_debug("PM: free some memory\n");
shrink_all_memory(FREE_PAGE_NUMBER - free_pages);
if (nr_free_pages() < FREE_PAGE_NUMBER) {
error = -ENOMEM;
printk(KERN_ERR "PM: No enough memory\n");
}
}
if (!error)
return 0;

Thaw:
suspend_thaw_processes();
usermodehelper_enable();
Finish:
pm_notifier_call_chain(PM_POST_SUSPEND);
pm_restore_console();
return error;
}
Suspend Devices
For now, all the other process(process/workqueue/kthread) is stoped, they may have locked semaphore, if you waiting for them in driver's suspend function, it will a dead lock. And then, kernel will free some memory for later use. Finally, it will call suspend_devices_and_enter() to suspend all devices, in this function, first will call suspend_ops->begin() if this machine have this function, device_suspend() in driver/base/power/main.c will be called, this function will call dpm_suspend() to all all device list and their suspend() callback. After suspend devices, it will call the suspend_ops->prepare() to let machine do some machine related prepare job(could be empty on some machine), it will disable nonboot cpus to avoid race conditions, so , after that, it will only one cpu will running. suspend_ops is a machine related pm op, normally it registed by arch/xxx/mach-xxx/pm.c And then, is suspend_enter() will be called, here will disable arch irqs will suspend, call device_power_down(), this message will call each of suspend_late() callback, thi will be the last call back before system hold, and suspend all system devices, I guess it means, all devices under /sys/devices/system/*, and then it will call suspend_pos->enter() to let cpu going to a power save mode, system will stop here, aka, the code executing stop here.
/**
* suspend_devices_and_enter - suspend devices and enter the desired system
* sleep state.
* @state: state to enter
*/
int suspend_devices_and_enter(suspend_state_t state)
{
int error, ftrace_save;

if (!suspend_ops)
return -ENOSYS;

if (suspend_ops->begin) {
error = suspend_ops->begin(state);
if (error)
goto Close;
}
suspend_console();
ftrace_save = __ftrace_enabled_save();
suspend_test_start();
error = device_suspend(PMSG_SUSPEND);
if (error) {
printk(KERN_ERR "PM: Some devices failed to suspend\n");
goto Recover_platform;
}
suspend_test_finish("suspend devices");
if (suspend_test(TEST_DEVICES))
goto Recover_platform;

if (suspend_ops->prepare) {
error = suspend_ops->prepare();
if (error)
goto Resume_devices;
}

if (suspend_test(TEST_PLATFORM))
goto Finish;

error = disable_nonboot_cpus();
if (!error && !suspend_test(TEST_CPUS))
suspend_enter(state);

enable_nonboot_cpus();
Finish:
if (suspend_ops->finish)
suspend_ops->finish();
Resume_devices:
suspend_test_start();
device_resume(PMSG_RESUME);
suspend_test_finish("resume devices");
__ftrace_enabled_restore(ftrace_save);
resume_console();
Close:
if (suspend_ops->end)
suspend_ops->end();
return error;

Recover_platform:
if (suspend_ops->recover)
suspend_ops->recover();
goto Resume_devices;
}
Resume
If the system wake up by interrupt or other event, the code executing will be continue. The first thing system resume is resume the devices under /sys/devices/system/, and enable irq, and then, it will enable nonboot cpus, and call suspend_ops->finish() to let machine know it will start resume, suspend_devices_and_enter() function later will will call every device 's resume() fucntion to resume devices, resume the console, and finally, call the suspend_ops->end(). Let's return to enter_state() function, after suspend_devices_and_enter() returns, the devices is running, but user space process and task is still freezed, enter_state will later call suspend_finish(), it will thaw the processes and enable user mode helper, and notify all pm they are exit from a suspend stage, and resume the console. This is a stardard linux suspend and resume sequence.
Android Suspend
In android patched kernel, going to request_suspend_state() in kernel/power/earlysuspend.c (since android add the Early suspend & wakelock feather in kernel). For detail understand that, let first introduct serval new feather android imported.
Files:
linux_source/kernel/power/main.c
linux_source/kernel/power/earlysuspend.c
linux_source/kernel/power/wakelock.c
Early Suspend
Early suspend is a mechanism that android introduced into linux kernel. This state is btween really suspend, and trun off screen. After Screen is off, several device such as LCD backlight, gsensor, touchscreen will stop for battery life and functional requirement.
Late Resume
Late resume is a mechinism pairs to early suspend, executed after the kernel and system resume finished. It will resume the devices suspended during early suspend.
Wake Lock
Wake lock acts as a core member in android power management system. wake lock is a lock can be hold by kernel space ,system servers and applications with or without timeout. In an android patched linux kernel (referenced as android kernel below) will timing how many and how long the lock have. If there isn't any of wake lock prevent suspend(WAKE_LOCK_SUSPEND), android kernel will call linux suspend (pm_suspend()) to let entire system going to suspend.
Android Suspend
when user write "mem"/"stanby" to /sys/power/state the state_store() will called. And then will going to request_suspend_state(), this function will check the state, if the request is suspend it will queue the early_suspend_work -> early_suspend(),
void request_suspend_state(suspend_state_t new_state)
{
unsigned long irqflags;
int old_sleep;

spin_lock_irqsave(&state_lock, irqflags);
old_sleep = state & SUSPEND_REQUESTED;
if (debug_mask & DEBUG_USER_STATE) {
struct timespec ts;
struct rtc_time tm;
getnstimeofday(&ts);
rtc_time_to_tm(ts.tv_sec, &tm);
pr_info("request_suspend_state: %s (%d->%d) at %lld "
"(%d-%02d-%02d %02d:%02d:%02d.%09lu UTC)\n",
new_state != PM_SUSPEND_ON ? "sleep" : "wakeup",
requested_suspend_state, new_state,
ktime_to_ns(ktime_get()),
tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,
tm.tm_hour, tm.tm_min, tm.tm_sec, ts.tv_nsec);
}
if (!old_sleep && new_state != PM_SUSPEND_ON) {
state |= SUSPEND_REQUESTED;
queue_work(suspend_work_queue, &early_suspend_work);
} else if (old_sleep && new_state == PM_SUSPEND_ON) {
state &= ~SUSPEND_REQUESTED;
wake_lock(&main_wake_lock);
queue_work(suspend_work_queue, &late_resume_work);
}
requested_suspend_state = new_state;
spin_unlock_irqrestore(&state_lock, irqflags);
}
Early Suspend
in early_suspend(): It will first check was the state still suspend (in case the suspend request was canceled during the time), if abort, this work will quit. If not, this func will call the all of registered early suspend handlers, and call suspend() of these handlers. And then, sync file system, and most important, give up a main_wake_lock, this wake lock is used by wakelock self and early suspend. This wake lock is not a timeout wake lock, so, if this lock is holded, wake lock will going to suspend even these was none of wake lock actived. During this time, the system suspend was not called. Because of early suspend give up the main_wake_lock, so the wake lock can decide if going to suspend the system.
static void early_suspend(struct work_struct *work)
{
struct early_suspend *pos;
unsigned long irqflags;
int abort = 0;

mutex_lock(&early_suspend_lock);
spin_lock_irqsave(&state_lock, irqflags);
if (state == SUSPEND_REQUESTED)
state |= SUSPENDED;
else
abort = 1;
spin_unlock_irqrestore(&state_lock, irqflags);

if (abort) {
if (debug_mask & DEBUG_SUSPEND)
pr_info("early_suspend: abort, state %d\n", state);
mutex_unlock(&early_suspend_lock);
goto abort;
}

if (debug_mask & DEBUG_SUSPEND)
pr_info("early_suspend: call handlers\n");
list_for_each_entry(pos, &early_suspend_handlers, link) {
if (pos->suspend != NULL)
pos->suspend(pos);
}
mutex_unlock(&early_suspend_lock);

if (debug_mask & DEBUG_SUSPEND)
pr_info("early_suspend: sync\n");

sys_sync();
abort:
spin_lock_irqsave(&state_lock, irqflags);
if (state == SUSPEND_REQUESTED_AND_SUSPENDED)
wake_unlock(&main_wake_lock);
spin_unlock_irqrestore(&state_lock, irqflags);
}
Late Resume
After all the kernel resume is finished, the user space process and service is running, the wake up of system for these reasons:
->
In Calling, the modem will send command to rild (RING command), and rild will send message to WindowManager and Application to deal with in call event, PowerManagerSerivce also will write "on" to interface to let kernel execute late resume.
->
User Key EventWhen system waked by a key event, such as a power key, or menu key, these key event will send to WindowManager, and it will deal with it, if the key is not the key can wake up system, such as return key/home key, the WindowManager will drop the wake lock to let system going to suspend again. if the key is a wake key, the WindowManager will RPC PowerManagerSerivce interface to execute late resume.
->
Late Resume will call the resume func in list of early suspend devices.
static void late_resume(struct work_struct *work)
{
struct early_suspend *pos;
unsigned long irqflags;
int abort = 0;

mutex_lock(&early_suspend_lock);
spin_lock_irqsave(&state_lock, irqflags);
if (state == SUSPENDED)
state &= ~SUSPENDED;
else
abort = 1;
spin_unlock_irqrestore(&state_lock, irqflags);

if (abort) {
if (debug_mask & DEBUG_SUSPEND)
pr_info("late_resume: abort, state %d\n", state);
goto abort;
}
if (debug_mask & DEBUG_SUSPEND)
pr_info("late_resume: call handlers\n");
list_for_each_entry_reverse(pos, &early_suspend_handlers, link)
if (pos->resume != NULL)
pos->resume(pos);
if (debug_mask & DEBUG_SUSPEND)
pr_info("late_resume: done\n");
abort:
mutex_unlock(&early_suspend_lock);
}
Wake Lock
Let's see how the wake lock mechinism run, we will focus on file wakelock.c. wake lock have to state, lock or unlock. The Lock have two method:
->
Unlimited LockThis type of lock will never unlock until some one call unlock
->
Wake Lock with TimeoutThis type of lock is alloc with a timeout, is the time expired, this lock will automatic unlock.
Also have two type of lock:
1. WAKE_LOCK_SUSPENDThis type of Lock will prevent system going to suspend.
2. WAKE_LOCK_IDLEThis type of Lock not prevent system going to suspend, not a lock can make system wake, I can't figure out why this lock exist.In wake lock functions, there was 3 enter pointer can call the suspend() workqueue:
1. In wake_unlock(), if there was none of wake lock after unlock, the suspend started.
2. after the timeout timer expired, the callback of timer will be called, in this function, it will check if there no of wake lock, system goto suspend.
3. In wake_lock(), if add lock success, it will check if there was none of wake lock, if none of wake lock, it will going to suspend. I think the way check here is unnessary at all, the better way is let wake_lock() wake_unlock() to be atomic, since this check add here also have chance missing the unlock.

Wakelock debug
There is a very useful way to enable wake lock's debug information in runtime as below, it will print all wake lock acquire and release information in your console, it's very useful while debugging the suspend/resume issue on android.
echo 15 > /sys/module/wakelock/parameter/debug_mask
Suspend
If the wake lock call the suspend workqueue, the suspend() will be called, this function check wake lock,sysc file system, and then call the pm_suspend()->enter_state() to going standard linux suspend sequence.
static void suspend(struct work_struct *work)
{
int ret;
int entry_event_num;

if (has_wake_lock(WAKE_LOCK_SUSPEND)) {
if (debug_mask & DEBUG_SUSPEND)
pr_info("suspend: abort suspend\n");
return;
}

entry_event_num = current_event_num;
sys_sync();
if (debug_mask & DEBUG_SUSPEND)
pr_info("suspend: enter suspend\n");
ret = pm_suspend(requested_suspend_state);
if (current_event_num == entry_event_num) {
wake_lock_timeout(&unknown_wakeup, HZ / 2);
}
}
Different Between Standard Linux Suspend
the pm_suspend() will call the enter_state() to going to a suspend() state, but it's not 100% same as standard kernel suspend sequence:
-> When freezing process, android will check if there was any of wakelock, if have, the suspend sequence will be interrupted.
->
In suspend_late callback, this callback will have a final check of wake lock, if some driver or freezed have the wake lock, it will return an error, this will make system going to resume. This could a problem in some situation. But this check is can't avoid, since the caller of wake_lock() normally not check the return value. So maybe some process start freezing without wake lock, but acquire some wake lock during the freezing, (I'm sure would this happen).
If the pm_suspend() success, the log after that will not seen until system resume success. some times, folks said can't see the log printed in suspend, some times is some error on resume, so the log will never been seen. So the suspend error is hard to debug. The log during suspend can print to console by add "no_console_suspend" to kernel command line , thanks kasim.
A more detailed about linux suspend please see http://kerneltrap.org/node/14004
Some tricks and tips on android suspend/resume
1. No_console_suspend the
In the kernel boot parameters inside with no_console_suspend, this is the most basic, because the kernel out to the console the suspend out of what was going on from the serial port can not see. The majority of crashes in the suspend / resume when you can see the kernel Panic is the information through the serial port, this will know what went wrong. Wrong, because sometimes resume or suspend the very back of the console of the error does not add this parameter can not see.
2. Initcall_debug
Few people may know, in fact, sometimes you do not know which driver when in the suspend / resume error when, very confused thought where to add some debug information to see where the driver, in fact, some time increase in inappropriate will not see a lot of useful information. In fact, the kernel itself has such a function (though not very user-friendly).
echo 1> / sys / module / kernel / parameters / initcall_debug
echo 9> / proc / sys / kernel / the printk
The first command is Open initcall_debug, this is the kernel will also be inside the startup parameters plus initcall_debug turned on by default this parameter, so you can debug the system startup.
KERN_DEBUG level of this information, so it is necessary to improve the level of printk can be seen, otherwise the suspend / resume is dead, you will not have a chance to see this information.
3. Suspend_test
This method can be used the rtc such software to do the cycle of the suspend / resume for Android this is not quite enough, (you have to simulate a POWER_KEY up enough), but still have some use for the stability of the debugging Driver the. Do not think that suspend several times you can, then there can be thousands of times the test. This suspend is 5 seconds using an RTC wake-up, then for the Android, 5 seconds will automatically go on forever, but for general-purpose Linux, you have to write a small script to make his will sleep on, and perhaps more useful this tool rtcwakeup (google rtcwakeup).
How to use:
Compile a kernel, the make menuconfig, selected
CONFIG_PM_DEBBUG = y
CONFIG_PM_TEST_SUSPEND = y
These two options
Programming the new kernel, and then open the stuff you need to test, such as WIFI, 3G
echo "core"> / sys / power / pm_test
echo "mem"> / sys / power / state
In this way, it will cycle sleep and wake-up call.
4. Wakelock
Turn the debugging of Android, the Android inside and Power related to the largest wakelock, sometimes encounter not sleep down, or sleep the last couple of chances, is the wakelock caused, or is caused by wakelock the user.
How to debug it, use two:
echo 15> / sys / module / wakelock / parameters / debug_mask
echo 15> / sys / module / userwakelock / parameters / debug_mask
15 F representing hexadecimal, in wakelock which is open all the debug information, at least for now. If the future is not enough, the estimated input 255.
So you can see the kernel and frameworks layers wakelock the operation, application and release. So look at the application and release pairs and whether it can be. Note: wakelock a timeout, the means that the number of milliseconds later, it will automatically release, For these wakelock, application and release may not. .
5. Power.0
Sometimes you'll see the system suspend to the last, and then encountered power.0 suspend failed, then the entire system and resume it. This is the android's proprietary because power.0 is the android registered to suspend a callback, it will check the CPU enters suspend no wakelock, this time there is no release wakelock, then it will return - The EBUSY then cause the entire suspend to fail. Debug this problem is to debug messages wakelock above open, and then see which guy to go applied wakelock, and then kill it.
Erroneous error message something like this:
pm_noirq_op (): platform_pm_suspend_noirq +0 x0/0x38 returns -11
PM: Device power.0 failed to the suspend late: error -11
6. Earlysuspend the
Almost forget about this guy, the android inside a pm the big stuff, the same can be added:
echo 15> / sys / module / earlysuspend / parameters / debug_mask
Print out the debug information, such as that earlysuspend want to call the class.
7 suspend / resume time.
Sometimes you want to debug the suspend / resume time is too slow. One approach is to use initcall_debug, printk timestamp marked, and then see which is very slow.
But this will make your life very boring :)
I have a patch designed to debug this problem, but upstream does not accept, that have to use this grueling the job, but if you want to use can be down under, playing up with it.
Address here: http://www.spinics.net/lists/linux-pm/msg24063.html
The usage is marked with this PATCH KERNEL inside select PM_DEBUG, SUSPEND_DEVICE_TIME_DEBUG these two options.
Then
microsecond echo> / sys / power / device_suspend_time_threshold
Such as
echo 50000> / sys / power / device_suspend_time_threshold
Note here microseconds Oh. . . It will slow driver when the suspend / resume the fight, and then you go to get rid of it. . .

To see if target is going to deep sleep
Perform cat on the file after waking up from deepsleep.

mkdir debugfs
mount -t debugfs none /debugfs
cat /debugfs/suspend/deesleep_count

Thursday, December 1, 2011

Order in which linux device drivers are probed!

The oder in which the device driver are initialized depends on the level of initcall in the driver. The levels range from 0-7.

#define pure_initcall(fn) __define_initcall("0",fn,0)

#define core_initcall(fn) __define_initcall("1",fn,1)
#define core_initcall_sync(fn) __define_initcall("1s",fn,1s)
#define postcore_initcall(fn) __define_initcall("2",fn,2)
#define postcore_initcall_sync(fn) __define_initcall("2s",fn,2s)
#define arch_initcall(fn) __define_initcall("3",fn,3)
#define arch_initcall_sync(fn) __define_initcall("3s",fn,3s)
#define subsys_initcall(fn) __define_initcall("4",fn,4)
#define subsys_initcall_sync(fn) __define_initcall("4s",fn,4s)
#define fs_initcall(fn) __define_initcall("5",fn,5)
#define fs_initcall_sync(fn) __define_initcall("5s",fn,5s)
#define rootfs_initcall(fn) __define_initcall("rootfs",fn,rootfs)
#define device_initcall(fn) __define_initcall("6",fn,6)
#define device_initcall_sync(fn) __define_initcall("6s",fn,6s)
#define late_initcall(fn) __define_initcall("7",fn,7)
#define late_initcall_sync(fn) __define_initcall("7s",fn,7s)

Ref: include/linux/init.h

Below is the issue that I faced.

I have an mfd device and the count of the clients goes upto 14. Now that I am interested in only 3 among them(say x, y and z) and the order that I want them to be probed in is x->y->z.
In my mfd core driver, the order in which I add the devices also suits my requirement, i.e x->y->z.
The level of initcall in all the 3 drivers are same, say subsys_initcall_sync(). Also all the 3 drivers(x, y and z) are in the same folder.

The expected result would be 'x' should be probed first followed by 'y' and 'z'.
But no, the order that I got is x->z->y.

I tried altering the order in which the mfd devices are added but no use. The actual problem was in the Makefile, the order in which linking had happened.
In makefile
obj-$(CONFIG_XXXX) += x.0 z.0 y.0
Here is where the issue was, later I changed the order in makefile and things started working as expected.

So remember the order in which linking happens also matters!

Saturday, November 26, 2011

Notifiers in Linux kernel

Adding new notifiers and calling the notifier call chain.
Assume that are two drivers and one driver will have to notify the other.
Driver-1
/* atomic notifier initialization */
ATOMIC_NOTIFIER_HEAD(simple_notifier_list);

unsigned long val;
struct device_info *di;
/* later in some part of the driver call the notifier call chain */
atomic_notifier_call_chain(&simple_notifier_list, val, (void *)di);
Driver-2
static int notify_call_handler(notifier_block *n, unsigned long event, void *data)
{
}
static struct notifier_block simple_nb = {
.notifier_call = simple_notify_handler;
}
atomic_notifier_chain_register(&simple_notifier_list, &simple_nb)

Thursday, May 19, 2011

iotop: Per Process I/O Usage | Linux Magazine

Monitoring the IO usage of individual processes on a system has been much easier with iotop. It also includes some interesting and very useful performance monitoring features.

iotop: Per Process I/O Usage | Linux Magazine


Friday, April 29, 2011

Receive Packet Steering (RPS) on Linux 2.6.35

This patch implements software receive side packet steering (RPS). RPS distributes the load of received packet processing across multiple CPUs. Problem statement: Protocol processing done in the NAPI context for received packets is serialized per device queue and becomes a bottleneck under high packet load. This substantially limits pps that can be achieved on a single queue NIC and provides no scaling with multiple cores. (lwn.net: Software receive packet steering)


If you want to find out whether RPS is working, you have to look at /proc/softirqs instead (eg. with watch -n1 cat /proc/softirqs):
                CPU0       CPU1
       HI:          0          0
    TIMER:  480622794  476948579
   NET_TX:   25311134   27075847     <-----
   NET_RX: 1388399338 4191697027     <-----
    BLOCK:    4632803          3
 BLOCK_IOPOLL:          0          0
  TASKLET:         21          4
    SCHED:  154913375  158601463
  HRTIMER:    1576760    2361409
      RCU:  421549961  407634645 

Enabling RPS specifically for a particular interface:

# cat /sys/class/net/eth0/queues/rx-0/rps_cpus 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,00000000  # echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus # cat /sys/class/net/eth0/queues/rx-0/rps_cpus 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000, 00000000,00000000,00000000,00000000,00000000,00000000,0000000f