Mr Tillerson, 64, faced tough questions from the Senate Foreign Relations Committee on Wednesday.
In a heated exchange, Mr Rubio grilled him about whether intelligence reports about Russia's involvement in hacks on the US election were accurate and if Mr Putin had directed the attacks.
Mr Tillerson said he had no inside information on the detailed intelligence about Russia's hacking, but he had read the declassified US report released last week on the issue.
The Florida senator suggested that Mr Putin was responsible for war crimes because of Moscow's support for Syrian President Bashar Assad and bombing of Aleppo.
But the Texan multimillionaire told Mr Rubio he would not describe Mr Putin as a war criminal.
"I would not use that term," Mr Tillerson said. "Those are very, very serious charges to make and I'd want to have much more information before reaching that conclusion," he added.
The Florida senator - who was one of Mr Trump's rivals for the Republican nomination - said he had "serious concerns" about Mr Tillerson as America's top diplomat.
Tillerson in jeopardy? - Anthony Zurcher, BBC North America Reporter
It looks like secretary of state nominee Rex Tillerson has his work cut out for him if he wants to win the support of Florida Republican Senator Marco Rubio.
In a blistering 10 minutes of questioning, the former presidential candidate, who was once belittled by Donald Trump as "little Marco", peppered the president-elect's chosen man with a series of questions about his views on Russia.
Mr Tillerson's answers clearly didn't satisfy Mr Rubio, who said they were "discouraging", as he shook his head throughout the exchange.
Given the narrow majority Republicans have in the Senate, if Mr Rubio - and a handful of other Republicans - fail to back Mr Tillerson, his nomination could be on shaky ground.
In fact, the Florida senator, through his position on the confirmation committee, could join forces with Democrats to delay or even derail Mr Tillerson's bid to be secretary of state before it reaches a vote in the full Senate.
With Mr Trump's relations with Russia in the US political spotlight over the past few days, it's Mr Tillerson's nomination that may be in the greatest jeopardy.
While Mr Tillerson was grilled by senators in Washington DC, up in New York Mr Trump was rejecting claims that Russian intelligence agencies have compromising information about the president-elect.
Russia has called the allegations "pulp fiction" and a "clear attempt to damage relations".
In his Senate statement, Mr Tillerson warned that Americans should be "clear-eyed about our relationship with Russia".
"Russia today poses a danger, but it is not unpredictable in advancing its own interests. It has invaded Ukraine, including the taking of Crimea, and supported Syrian forces that brutally violate the laws of war.
"Our Nato allies are right to be alarmed at a resurgent Russia," he continued, adding: "But it was in the absence of American leadership that this door was left open and unintended signals were sent.
It is Mr Tillerson's connections to Russia that have drawn the most flak in recent months.
He has forged multi-billion-dollar deals with Russia's state oil company, Rosneft, spoken out against international sanctions imposed on Moscow and in 2013 was awarded an Order of Friendship by the Kremlin.