removing baseline signal using fourier transforms

Question

I have timeseries data for many terms an example of which is below:

term1 = [0.0, 0.0, 0.0, 0.0, 2.2384935833581433e-06, 3.938767914008819e-06, 0.0, 0.0, 1.1961851263949013e-06, 0.0, 2.278384397623645e-06, 1.100158422812885e-06, 0.0, 1.095521835393462e-06, 0.0, 0.0, 1.6933152148605343e-06, 0.0, 8.460737945563612e-07, 8.949410770794851e-07, 0.0, 2.8698467119209605e-06, 0.0, 0.0, 0.0, 3.9163008188985015e-06, 2.2244961516216576e-06, 0.0, 0.0, 1.9407903674692482e-06, 0.0, 0.0, 0.0, 0.0, 9.514657329616274e-07, 1.94463053478312e-06, 0.0, 0.0, 0.0, 2.0373216961518047e-06, 1.8835690620014428e-06, 0.0, 0.0, 0.0, 0.0, 9.707946148081127e-07, 0.0, 0.0, 1.6121985390256838e-06, 1.9547361301697883e-06, 0.0, 2.2876018840689116e-06, 2.208826914114183e-06, 1.9640500282823203e-06, 0.0, 2.6234669115235785e-06, 0.0, 0.0, 0.0, 1.986207773222741e-06, 1.049193537387487e-06, 1.090723073046815e-06, 0.0, 1.0257546476943088e-06, 9.179053033814713e-07, 0.0, 0.0, 0.0, 0.0, 9.335621182897889e-07, 0.0, 0.0, 0.0, 0.0, 2.1267500494469387e-06, 2.215050381320923e-06, 2.163720040591388e-06, 1.937729136470388e-06, 1.6037643556956889e-06, 1.313906783569333e-06, 0.0, 1.0064645216223805e-06, 1.876346865234201e-06, 9.504447606257348e-07, 2.017974095266539e-06, 0.0, 2.120782823355757e-06, 0.0, 0.0, 0.0, 0.0, 9.216394491176685e-07, 0.0, 0.0, 1.0401357169083422e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0089962853658684e-06, 1.8249773702806084e-06, 0.0, 1.2890950295073852e-06, 5.42812725267281e-06, 1.9185480428411778e-06, 2.6955316172381044e-06, 0.0, 0.0, 1.0070239923466176e-06, 0.0, 1.021152145542773e-06, 9.919749228739498e-07, 1.9293082175989564e-06, 9.802489636317832e-07, 1.0483850676418046e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 1.9369409504181854e-06, 0.0, 4.619620451983665e-06, 0.0, 6.0795324434248845e-06, 0.0, 1.5312669396405198e-06, 1.2797051559320733e-06, 1.1002903666277531e-06, 0.0, 1.0054768323055684e-06, 2.060260561153169e-06, 1.0898719291496056e-06, 3.4605907920600203e-06, 3.3500051925080486e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 6.5521496510980315e-06, 0.0, 0.0, 0.0, 3.01862187836765e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.849817053093449e-06, 6.5552277941658475e-06, 1.985771944021089e-06, 1.010233667047188e-06, 9.802307070992228e-07, 5.605931075077432e-06, 3.651067480854715e-06, 0.0, 0.0, 2.9476960807432912e-06, 1.834478659509754e-06, 0.0, 0.0, 0.0, 0.0, 3.3801712394749917e-06, 0.0, 2.2884970981856794e-06, 1.02014792144861e-06, 2.906143199237428e-06, 9.807873564740302e-07, 0.0, 2.106593638087213e-06, 3.0329622335542676e-06, 2.9093758515985565e-06, 0.0, 2.12762335960239e-06, 9.614820669172289e-07, 9.264114341404848e-07, 0.0, 0.0, 9.073611487918033e-07, 0.0, 0.0, 0.0, 6.0360958532021484e-06, 0.0, 4.553288270957079e-06, 2.0712553257152562e-06, 3.292603824030081e-06, 2.690786880261329e-06, 2.301011409565074e-06, 2.029661472762958e-06, 0.0, 9.657114492818003e-07, 9.948942029504583e-07, 1.028682761437152e-06, 2.0694207898151387e-06, 3.845369982272845e-06, 9.048250701691842e-07, 1.7726379156614332e-06, 0.0, 9.238711680133629e-07, 9.231112912203808e-07, 9.422814896339613e-07, 0.0, 1.2123519263665934e-06, 0.0, 0.0, 2.1675188628329036e-06, 0.0, 4.498718989767663e-06, 0.0, 0.0, 2.650273839544471e-06, 1.1954029583832415e-06, 4.180999656112778e-06, 1.9036523473937095e-06, 9.75877289286136e-07, 0.0, 2.093618232902467e-06, 1.032899928523325e-06, 0.0, 4.473312219299659e-06, 8.762705923589204e-07, 0.0, 0.0, 1.792797436299666e-06, 0.0, 0.0, 1.1974513445582422e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.1404264054329915e-06, 3.061324451410658e-06, 9.84829683554526e-07, 2.932895354293759e-06, 2.0897069394988045e-06, 0.0, 2.128187093183736e-06, 0.0, 4.686861415132188e-06, 6.37755683086446e-06, 1.8420463661490824e-06, 2.8347403094402523e-06, 1.9033842171380715e-06, 6.909144746582441e-06, 0.0, 0.0, 1.5479612576256442e-06, 5.621978186724636e-06, 2.087185930697078e-06, 1.3168406359462377e-05, 1.9676130885622652e-05, 1.9988766313331908e-05, 3.1801079228204546e-05, 3.322824899588385e-05, 2.0358501231090545e-05, 1.2383952049337664e-05, 1.8052256532066507e-05, 7.770543617518302e-06, 9.226179797741636e-06, 4.400430362089412e-06, 4.333084180992927e-06, 7.477274426653279e-06, 3.0526428255261993e-06, 4.952368123389242e-06, 1.2578584707962998e-05, 0.0, 2.121750274236223e-06, 0.0, 2.38940918273843e-06, 0.0, 1.5693511988273807e-06, 0.0, 0.0, 4.520448247648237e-06, 4.0303440122522456e-05, 2.8660979509446863e-05, 2.4793768971660722e-05, 3.957070185234852e-05, 2.64488881248099e-05, 6.428381095168035e-05, 5.6557662521419976e-05, 6.855540059858658e-05, 7.079288025889968e-05, 7.135683422742382e-05, 5.5663480860112103e-05, 8.088436527379357e-05, 7.142268494354861e-05, 8.243171356847987e-05, 7.658173644233611e-05, 5.4275733753644613e-05, 2.7329513031804995e-05, 1.8666856995404658e-05, 2.5061514626811264e-05, 9.707359513272993e-06, 2.233654188450612e-05, 2.0577084330035857e-05, 6.037067595033506e-05, 5.358585847760433e-05, 6.353114888415205e-05, 4.913406130358561e-05, 6.253876100291326e-05, 5.783647108547192e-05, 5.29265883017118e-05, 4.295770587763158e-05, 0.00012513639867455526, 0.0001264425725280477, 0.00010075697417828198, 7.700585441944497e-05, 6.390017630639553e-05, 6.862379380485504e-05, 8.118867124374998e-05, 8.928305705187346e-05, 8.923668314113125e-05, 5.0862818355003976e-05, 2.5192448399293734e-05, 1.9491995287268695e-05, 1.1397180337584482e-05, 1.8548131739430545e-05, 2.8274146120787152e-05, 2.9861740143137274e-05, 5.749201435920551e-05, 8.676081065218611e-05, 0.00011692016691003383, 6.18107213073443e-05, 8.31986307882476e-05, 5.661490072734421e-05, 6.637785526376392e-05, 6.189842468509176e-05, 5.077848495281155e-05, 3.7630726455798414e-05, 6.325167842846687e-05, 7.447442335517917e-05, 7.881778491014126e-05, 8.347575938861497e-05, 6.553610066345062e-05, 6.209221186256924e-05, 4.671174184109858e-05, 4.583301504850661e-05, 2.9423292949863758e-05, 1.9969520206001368e-05, 1.3386836054765546e-05, 1.0233804045678584e-05, 2.3371876153986385e-05, 3.701784260013326e-05, 2.6804842191646374e-05, 3.729558727386808e-05, 7.011179438698544e-05, 4.616049584765358e-05, 6.019787395273405e-05, 8.312188292939014e-05, 6.281596430043117e-05, 6.370630077282333e-05, 6.169767733530766e-05, 6.099512039036877e-05, 7.192322709245217e-05, 6.727547574464268e-05, 4.891125624919348e-05, 8.775231227342841e-05, 9.349358010749929e-05, 4.85363097385816e-05, 4.475820776946539e-05, 1.9528637281926147e-05, 1.5243002033035396e-05, 1.4322461630125293e-05, 1.0492122514416176e-05, 1.1956759574674148e-05, 1.5232250274180506e-05, 3.394641638997643e-05, 2.6115894879792267e-05, 4.868559048521277e-05, 5.612535494090208e-05, 3.269545148571978e-05, 4.967751016319062e-05, 4.8382804751191425e-05, 5.1860846075881435e-05, 4.4034258653232213e-05, 5.362193446127224e-05, 6.213052893181175e-05, 8.561827093901839e-05, 5.877682625663455e-05, 0.0, 0.0, 7.105805443046969e-05, 0.0, 0.0, 2.31393994554528e-05, 7.05044594070575e-06, 2.21491300929156e-05, 4.926848615186025e-06, 1.0752514744385843e-05, 1.4745260873155369e-05, 1.976297604068538e-05, 3.094705732168692e-05, 5.068338091939653e-05, 2.655137469742496e-05, 3.0142790705685793e-05, 3.89279249469607e-05, 6.264176821226988e-05, 3.598536226187379e-05, 4.430195278344506e-05, 2.7501831818440764e-05, 1.7243328268903956e-05, 1.2049184772240285e-05, 2.1016880758625327e-05, 3.411070201956675e-05, 3.1893789428697184e-05, 1.8509911029027654e-05, 3.920735117199027e-05, 3.700840501998454e-05, 8.529330234343347e-06, 1.1007881643256571e-05, 4.661265813344272e-06, 7.306007242688513e-06, 2.6772256446090046e-06, 3.0075821145106816e-06, 6.713527085725027e-06, 2.204123915846549e-05, 7.880065404542858e-06, 4.3539870647002475e-05, 6.0898558226633984e-05, 7.956054903697144e-05, 4.80968670903199e-05, 3.476307626484116e-05, 3.233622280581405e-05, 4.097520999795124e-05, 1.6048981491512094e-05, 3.4725910431663494e-05, 2.3840743831207534e-05, 4.194630872483221e-05, 3.472531193096608e-05, 2.9240209403218155e-05, 2.5871727972711297e-05, 1.1918039641386187e-05, 1.2189485552920143e-05, 8.254477280067191e-06, 5.343416003103456e-06, 0.0, 4.795714549478586e-06, 6.705621859254362e-06, 9.484831383410081e-06, 2.503719812292549e-05, 1.9037212038371403e-05, 2.448114104715256e-05, 3.2063674685728836e-05, 2.73499598297465e-05, 2.6255716088190032e-05, 2.930473870366029e-05, 2.490020970307041e-05, 2.4037259675477766e-05, 1.8683888229243836e-05, 9.573344744760269e-06, 2.01589736663327e-05, 2.8955116521484698e-05, 1.934869527601605e-05, 2.1111566182648825e-05, 1.0035410663340645e-05, 4.154485944681635e-06, 8.468739061212046e-06, 8.415088253238056e-06, 1.3883239181832948e-06, 0.0, 2.9995080806747692e-06, 1.6303266848611124e-06, 3.714448088730736e-06, 8.976418947425114e-06, 9.729566693747293e-06, 3.3588780313874236e-05, 1.7154466165266127e-05, 1.9646193877372823e-05, 9.475852684603824e-06, 9.763432041631274e-06, 2.5840349706066022e-05, 1.4272109443725072e-05, 2.262309793162043e-05, 1.733067926359468e-05, 8.405046389852468e-06, 1.6489619195801272e-05, 6.6721749177376435e-06, 5.2645543870584616e-06, 5.563468043439559e-06, 5.668953522517651e-06, 2.564151874715539e-06, 4.72535638047152e-06, 1.1322053548784465e-06, 4.683593955822e-06, 5.170243182388084e-06, 1.4458242427134072e-06, 5.110793484760465e-06, 8.06295555698897e-06, 1.7613618850094893e-05, 1.3702227753862316e-05, 1.2582942563061514e-05, 1.5863866870429223e-05, 5.763738591399926e-06, 5.010013765012819e-06, 3.355941190486578e-06, 1.2264709219075303e-05, 3.0533139142568385e-06, 5.2266756983622735e-06, 3.0845411025383717e-06, 7.013177761012944e-06, 1.5042033081191253e-05, 7.918060391926394e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0146814988996414e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 1.4253135689851767e-05, 0.0, 1.4218885523752648e-05, 0.0, 1.539361472861057e-05, 0.0, 3.981789947307646e-05, 0.0, 1.2433017120264575e-05, 1.2777756481516975e-05, 1.2764382267720154e-05, 0.0, 1.2131652695046646e-05, 0.0, 0.0, 1.9324418335008117e-05, 0.0, 5.3279343598486864e-05, 1.316118503310038e-05, 8.202637968370628e-06, 8.606938339893733e-06, 2.2898281255009e-06, 0.0, 3.510274573677153e-06, 1.6317872149471709e-06, 4.578600840631114e-06, 3.877291479264245e-06, 2.8741881616021876e-06, 0.0, 1.0729671296519832e-05, 4.808871405969733e-06, 4.534612698729401e-06, 4.3333188889370365e-06, 0.0, 2.743032696949748e-06, 4.019804235533729e-06, 1.8078426019917002e-06, 5.444991968636846e-06]

Each element is the combined signal for an hour and the list is over 24 days. Therefore, should have 24 days * 24 hours = 576 elements.

I am interested in the dynamics of the signal of each term over the course of days. However, confounding to this is the baseline changes of the signal within a day. I also have time series for basic terms that capture this baseline signal during a day such as the following.

baseline = [0.0056738537419516195, 0.005420397434626666, 0.005019676698052322, 0.004214006968007205, 0.004143451622795924, 0.00373395198248036, 0.0037080495988714344, 0.0036409523281401525, 0.003919898659196092, 0.004388163261294729, 0.004595501330006892, 0.005097033972892097, 0.0052221285817481335, 0.005184009325081863, 0.005273633551787361, 0.005053393415305126, 0.004444952439008902, 0.004552838940992971, 0.004808237374463801, 0.004895327691624783, 0.005059086256629757, 0.005598114319387153, 0.005952632334681949, 0.005717805004263755, 0.006126432469142252, 0.005592477569387059, 0.004920585487387107, 0.004318038669070883, 0.003877225571288378, 0.0036583898426795327, 0.0037336953886437474, 0.0037760782770061294, 0.0042338376814351954, 0.004192003050341723, 0.0046450557083186645, 0.004900468947653463, 0.005272546953959605, 0.005265723105151999, 0.0052304537716869855, 0.005121826744125637, 0.005224078793461002, 0.00501027352884918, 0.004871995260153345, 0.004863044486978714, 0.005310347911635811, 0.0058606870895965765, 0.00596470801561322, 0.005997180909289017, 0.00588291246890472, 0.005328610690842843, 0.004941976393965633, 0.004426509645673344, 0.0041172533679088375, 0.0038888190559989945, 0.003785501144341545, 0.0038683019610415165, 0.003826474222198437, 0.004178336738982966, 0.004574137078717032, 0.004854291797756379, 0.005216590267890586, 0.00514712218170792, 0.005217487414098377, 0.005239554740422529, 0.005138433888329476, 0.0050591314342241745, 0.005099277335119803, 0.00469742744667216, 0.005140145739820509, 0.005534156237221868, 0.006098190503066302, 0.00627542293362276, 0.005859315099582288, 0.0055863100804189264, 0.005193523620749424, 0.004680401455731111, 0.0041370327176107335, 0.003790198190936078, 0.0037143182477912154, 0.0037406926128218908, 0.0038838040372017974, 0.00413455625482474, 0.004309030576010342, 0.004768381364059312, 0.004905695025592956, 0.0050965947056771715, 0.005178951654634759, 0.005250840996574289, 0.005083679897873849, 0.0050438189257025106, 0.00465730975130931, 0.004511425103430987, 0.004631293617276186, 0.0049417738509291015, 0.005495036992426772, 0.0056409591251836465, 0.005487421237451456, 0.005093572532544252, 0.005043698439924855, 0.004837771685295603, 0.0038251289273366134, 0.003817852658627033, 0.003792612420533331, 0.003922716174790973, 0.00412646233748187, 0.004534488299255124, 0.004712687777471286, 0.005096266809417225, 0.00523394116440575, 0.005257672264041691, 0.005305086574696615, 0.005100654966986151, 0.004965826463906992, 0.005073115958176456, 0.00469441228683261, 0.004553136348768357, 0.004723653823501124, 0.004726168081415059, 0.005290955031631742, 0.005325956759690025, 0.005453676000994151, 0.005531903354394338, 0.005121462750913455, 0.004790546408707061, 0.004460326025284444, 0.003982093750443299, 0.0036151869988949132, 0.0035295702958713982, 0.003722662298606401, 0.004089779292755358, 0.004116707488056058, 0.004463311658604419, 0.004863245054602056, 0.005019950105663084, 0.005111292599872651, 0.0050328244675445916, 0.004886511461492081, 0.005017059119637564, 0.004997550003214928, 0.004989853142609061, 0.004888243576205561, 0.004801721031771264, 0.005142349216675533, 0.0053550501391269115, 0.00510410900976245, 0.005113311675603742, 0.004865951202283446, 0.004739388247627576, 0.004314592960862043, 0.003932197365205607, 0.0036889365827877003, 0.003444247563217489, 0.0033695476656641706, 0.003779678994400599, 0.004182362477080399, 0.004650999598571368, 0.004964528816231351, 0.005246502668776329, 0.005150211093436487, 0.0051813375657147505, 0.005326590813316477, 0.00501407415865325, 0.004920848192186853, 0.005020741681762219, 0.005108871853087233, 0.004991922013198609, 0.005551866678436957, 0.005681472655730911, 0.005624204122058199, 0.005202581478369662, 0.00490495583623749, 0.0043628317352519584, 0.0037568042368143423, 0.0035018559432594323, 0.0035627004864066413, 0.003560172130774401, 0.003604382929642445, 0.003782708492731446, 0.003958167037361377, 0.004405696805281344, 0.004888234197579893, 0.004849378554876764, 0.005035728295111269, 0.005150565049279978, 0.005104177573029002, 0.005540331228404623, 0.005146813504207926, 0.004991504807148932, 0.0050371760815936415, 0.005174258383207836, 0.005598418288045426, 0.0056576481335463774, 0.00561832393839059, 0.005408391628077189, 0.0052292710408241285, 0.004705309149638305, 0.003924934489565002, 0.003854606161156092, 0.0038935040219155712, 0.003830335124052002, 0.003746046574771941, 0.003865490274877053, 0.004168222873979538, 0.0045871293840885514, 0.004915772256778214, 0.005072434696646597, 0.00492522147976003, 0.004978792784547765, 0.004963870334948144, 0.004955409293231536, 0.004709890770618299, 0.004888202349958703, 0.0051805005663287775, 0.005568883603736712, 0.005781789868618008, 0.006061759631832967, 0.005730308168750368, 0.0055273545529884146, 0.005050318950400666, 0.004505314632141857, 0.0041320733921015994, 0.0037557073980650723, 0.0034979193552635043, 0.0037461620721961097, 0.0036352203964434373, 0.003974040173135196, 0.004094756199243869, 0.004649079406159152, 0.004920019940715673, 0.005231951964023264, 0.005121117845618645, 0.005064423379922766, 0.00498326981229982, 0.004871188222923238, 0.004660839287914527, 0.0047034466283560495, 0.004866548640835444, 0.005578880008506938, 0.0059683185805929845, 0.006061498706153822, 0.005800490254423062, 0.0054633509277901724, 0.004921961696040911, 0.004376719066835311, 0.00393610914724284, 0.0037954515471031775, 0.003581690980473693, 0.003563708289302751, 0.0037463007418473766, 0.00403278474399164, 0.004356886520045223, 0.004787462849992179, 0.005179338649547787, 0.005143654461390953, 0.005203417442834235, 0.005153892139635152, 0.005114303176192244, 0.00504646961230832, 0.00478839952880454, 0.004711338394289699, 0.004911682972324793, 0.005442432018950797, 0.005865476365139558, 0.006157467255298909, 0.005776991413458904, 0.00537648513923766, 0.005215877640811999, 0.004586994881879395, 0.00404235177861292, 0.0038098588593210615, 0.003611933103919232, 0.003782482344031445, 0.003847756732676113, 0.004015496451997738, 0.004222327790973872, 0.004767228509347478, 0.005026217727591916, 0.005032992226639765, 0.0051856184936032845, 0.005070660243331873, 0.005025667638424633, 0.004771111450073196, 0.0049169687623427365, 0.0, 0.004725137860724068, 0.00480564403797717, 0.004993865191923319, 0.005382243541508231, 0.005436232552738047, 0.005416886729676188, 0.004777014387860352, 0.0048255785043644925, 0.004081842852408802, 0.004090331218562488, 0.00378104976817826, 0.003521792464859018, 0.0036283065618489215, 0.003818665737661915, 0.003988803567300145, 0.004483523199147563, 0.004696601941747573, 0.005206918843848881, 0.005231253931233336, 0.005154439277447777, 0.005107271378732522, 0.004862372011026066, 0.005097539245443387, 0.004771922511620435, 0.004800155668906229, 0.004886324331150043, 0.005186594367994167, 0.0055550364814704704, 0.00565254113064783, 0.00542892074907446, 0.005216026402108949, 0.0050842262523550985, 0.004506330112231061, 0.004262871158699087, 0.004073705404217544, 0.003562133424289835, 0.003499455612234611, 0.0037587992642927636, 0.004170545895025578, 0.004646029409170125, 0.004941082109950799, 0.005336110809450001, 0.005238846272634943, 0.0051019151317224926, 0.004828998520466023, 0.00470819320853546, 0.004974373055097931, 0.004975308413634935, 0.005266317039295838, 0.005489162450620279, 0.005606273008057806, 0.00603476714807901, 0.0061970275556501725, 0.0058349840239690235, 0.005192678736923442, 0.004639151581343363, 0.004229911816211891, 0.003727661961919841, 0.00375780482393585, 0.0033937487713780225, 0.003400171769633621, 0.003719857252709842, 0.0037474521895174925, 0.004410321140619574, 0.00505109832021614, 0.00506160098731807, 0.005046922423918226, 0.005300710721177051, 0.005104647840739084, 0.004974276083656935, 0.004902745159619985, 0.005039594632444682, 0.005189007878086687, 0.005840559146565768, 0.005924790523904985, 0.006041782063467494, 0.006054874048959406, 0.005728511142370623, 0.005014567400775691, 0.004479858189014036, 0.004064222403658478, 0.0038690888337760544, 0.0038101713160671666, 0.0038192317788082945, 0.003855643888760465, 0.004151893395194348, 0.004439198456142054, 0.004868610511159107, 0.005164087705238066, 0.0052260812748906515, 0.005049306708959293, 0.005295364532855441, 0.004976241631407976, 0.005325257379808529, 0.004981215539676753, 0.004904617253355752, 0.005133080934624669, 0.005474999665809228, 0.006018281474269119, 0.0059556619441451936, 0.00582564335486158, 0.0057773567703702745, 0.005185870554701607, 0.004927387470357575, 0.004290471577704514, 0.003894605250504856, 0.0036579206650162693, 0.0037227880322444513, 0.0037587839308041025, 0.004025131552347727, 0.0043915455477435165, 0.004973183367291931, 0.005602412946227073, 0.005438255876982902, 0.005057281453194344, 0.0055819722968782305, 0.0052582960278547575, 0.0060302188495155494, 0.003969113083640037, 0.004874700151948723, 0.0048366059153241445, 0.005174590517957408, 0.005237240077942745, 0.005935388138900985, 0.006375850801552381, 0.006218749794135666, 0.005833520305137985, 0.005325978611613252, 0.00473056992525788, 0.0039874605990664344, 0.0038460789847597175, 0.003587065463944717, 0.00384212944765237, 0.004264645875837948, 0.004969973892903938, 0.005856983835337711, 0.006181231788159266, 0.006313470979891048, 0.006097287985997557, 0.005694104539336737, 0.005355534257732001, 0.005274420505031954, 0.004712403572544698, 0.004584515000959549, 0.004766412751530095, 0.0048104263193712886, 0.005309031929686986, 0.006042498279882524, 0.006496377367343072, 0.005619222170751848, 0.005418471293122766, 0.005015661629991529, 0.005062499505228742, 0.004308572994534354, 0.0038880894398937347, 0.003538125785331658, 0.0034843298748529253, 0.003774099147478583, 0.003896742470805163, 0.004541861762097922, 0.004553179667775172, 0.004948038015149709, 0.0050269339456022605, 0.00522398911361471, 0.005050975431726277, 0.005007174429180125, 0.004833758244552214, 0.004670604547693902, 0.00477521510651887, 0.004939453753268834, 0.005239435739336397, 0.005820798534429634, 0.006094069145690364, 0.005673509972509797, 0.005375844111251002, 0.005187640280456208, 0.00476628984541101, 0.004247493846603608, 0.003794806926377327, 0.003435122854871529, 0.003587919312587277, 0.003811897320196127, 0.0042459415490763925, 0.00460744683733153, 0.004807733730818607, 0.005155657515164588, 0.005405463068510853, 0.005224147724524333, 0.005351078308428722, 0.005384714635929638, 0.005362056525935763, 0.0051377016971353075, 0.004941059319359612, 0.004966034655341646, 0.005026256144832193, 0.005442607412384369, 0.0059898202401797275, 0.005612531062072142, 0.005603529527930128, 0.0051493731726657554, 0.004544820351700367, 0.004496920773323335, 0.00424357787751253, 0.0036690501594786006, 0.003700340743778253, 0.0038846659058119253, 0.004159671170598417, 0.004794839922729552, 0.005004852590193807, 0.005163099925195087, 0.005645338914676821, 0.005432262412191398, 0.0050802949835114155, 0.005169574505964038, 0.0052347116826927985, 0.0052757424822272225, 0.0056125420409050475, 0.005578375783486106, 0.005944651628427074, 0.006010407699526147, 0.0061534279769882615, 0.005756457061668538, 0.005283251628717022, 0.004694029423550289, 0.0042271372620665245, 0.003995084263772031, 0.003916612465121526, 0.00385882298694225, 0.0039353658124175695, 0.00403048536977438, 0.0039523025458470164, 0.004692943486212761, 0.005099144811322234, 0.005182029052264465, 0.005496327599559573, 0.0053953892408097875, 0.005256712315751134, 0.004628655585945719, 0.005255300089578979, 0.004727544165215228, 0.005365188522646431, 0.006321448616075385, 0.005962859901186893, 0.0064913517773445605, 0.006403310018717368, 0.005985231247570929, 0.005536676822123271, 0.005652983876148263, 0.0053962798830303575, 0.0036360246130896887, 0.0034235996705107084, 0.004421584551524996, 0.003810299791511898, 0.0038131330853627154, 0.0038483466362599773, 0.005120205311426739, 0.0048344210780759, 0.005090949889906456, 0.005557094917028417, 0.005276073619631902, 0.0056143238257037814, 0.005700457782933553, 0.00584351804652435, 0.004893880732421001, 0.005475919992851946, 0.005248580353868141, 0.005350058838515571, 0.006083169087767963, 0.005703392826945841, 0.006319084795547654, 0.005231157508317081, 0.005381213703447174, 0.005027572682644346, 0.0042202572347266884, 0.004068212855323105, 0.003991170422748069, 0.0037477607718658665, 0.004077183917326014, 0.00408925876065761, 0.004650332253801002, 0.004960348232472058, 0.005144796809267916, 0.00597460791635549, 0.005407754333445995, 0.005265714189536858, 0.005391654498789258, 0.00495731680894397, 0.005033086804203971, 0.00511026991441738, 0.005391897414595909, 0.006005653123816428, 0.0066265552258310415]

i think that a good way for me to extract out the signal I am interested in would be to do a spectral analysis on the timeseries for my terms. The high frequencies should be the daily patterns which I want to get rid of and the lower frequencies should be what I'm interested in. I want to somehow 'divide' my observed signal for a term by the baseline daily signal.

This is my baseline's original signal and this is my term's original signal and what i'm trying to do is get something like this in a general way without introducing artefacts. i.e remove the ups and downs that happen every day anyway and capture the general trend.

The naive way I thought of doing this is to first generate ffts for both using numpy(below).

baselines fft

term1s fft

and then create a filter like below

fft2 = fft(term1, n=t)
mgft2=abs(fft2)  
plot(mgft2[0:t/2+1])
bp  = fft2[:]
for i in range(len(bp)):
if i>=22:
    bp[i] = 0
ibp = ifft(bp)

but from what i understand that introduces artefacts, changes the magnitudes and I am not sure how to pick a cutting point. I was hoping for some guidance with respect to implementation in numpy on a better way to divide out my baseline frequencies from my term's frequencies. thanks

Answer 1

It looks like both your "baseline signal" and your timeseries data contain information overlapping in spectral content. As such, an FFT doesn't provide any extra help in removing the overlapping portions that end up in the same FFT frequency bins.

As an FFT is a linear operator, subtracting in the frequency domain isn't any different from subtracting directly in the time domain.

Answer 2

It's one thing to give the code, another to give an explanation. If you just want to look at the spectra, try the following:

import numpy as np
import matplotlib.pyplot as plt

hh = np.hanning(len(term1)) # Use a Hann window to deal with spectral leakage
St = np.fft.rfft(hh*term1)
Sb = np.fft.rfft(hh*baseline)

FSample = 1.0/24            # Sampling frequency is 1/24th of a day
deltaF = 1.0/(len(term1)*FSample)  # Frequency resolution is 1/capT = 1/(NumSamples*FSample)

faxis = deltaF*np.arange(len(St))

plt.plot(faxis, np.log10(np.abs(np.array([St, Sb]).T)))

See Wikipedia's explanation of spectral leakage and windowing for details on hh. The x-axis of the plot is calibrated in days. There are two thin peaks in the baseline spectrum at 1 and 2 day periodicity, and a broad peak around 1 day in the term1 spectrum.

It wasn't clear to me from your question if the baseline data already had the real data stripped out. If so, I think this shows that there isn't much structure in one that's distinguishable from the other.

Another thing to keep in mind is that Fourier analysis assumes the signal is stationary. Depending on the physics of what you're measuring this assumption may or may not be true. Certainly, the time-domain plot of term1 doesn't look at all stationary, with a wild change in character starting around day 12:

Having said all that, if you have a way to characterize the baseline data, you might be able to apply noise cancelation algorithms like the Widrow-Hoff LMS algorithm. Wikipedia presents a very theoretical overview, I'm not sure where to find a more practical application oriented explanation.

How did you come up with the term1/baseline separation in your example data?

Answer 3

Multiplying (or dividing) in the frequency domain is equivalent to convolving in the time domain. In other words a high-pass FIR filter would remove your low-frequency components directly without going into the frequency domain. If you do go to the frequency domain first be aware that simply removing some frequency components and converting back to time will introduce artifacts. FIR filter design is actually based on choosing filters you can multiply by which also meet your desired frequency specs.

All that said, it sounds like you already know the baselines and you could apply your adjustments directly from your known baselines to your data. The point of filtering would be that it would work without knowing the baselines.